Staff Backend Engineer, DataHub

• 8+ years building production-grade distributed systems
• Advanced Python and API design expertise
• Experience with high-scale data processing or integration frameworks
• Strong systems knowledge and distributed architecture experience
• Proven track record solving complex technical challenges
• Built and maintained online applications serving live traffic at scale (100+ QPS)
• Set up monitoring and alerting for services
• Designed indexing, storage, and data architectures to make large-scale data accessible to online services
• Designed and scaled distributed systems
• Hands-on experience developing in a tight loop with LLMs and applying best practices for scalable LLM development
• One of Java/Scala/Kotlin/C#/Go - very strong nice-to-have / borderline must-have
• Python/TypeScript/Node.js - nice-to-have
• AWS
• Kubernetes/Docker
• CI/CD deployment pipelines
• Microservice Architecture

Desirable

• As AI and data products become business-critical, enterprises face a metadata crisis:
• No unified way to track the complex data supply chain feeding AI systems
• Engineering teams struggling with data discovery, lineage, and governance
• Organizations needing machine-scale metadata management, not just human-browsable catalogs
• This is where infrastructure meets impact
• The metadata layer you'll build will directly power the next generation of AI systems at massive scale
• Your code will determine how safely and effectively thousands of organizations deploy AI, affecting millions of users worldwide
• We're looking for an exceptional Python engineer to lead development of DataHub's Platform framework – the core that connects diverse data systems and powers our metadata collection capabilities
• Scalable, fault-tolerant ingestion systems for enterprise-scale metadata
• Clean, intuitive APIs for our connector ecosystem
• Event-driven architectures for real-time metadata processing
• Schema mapping between diverse systems and DataHub's unified model
• Versioning systems for AI assets (training data, model weights, embeddings)