Staff Data Engineer, Duetto

• 7+ years building production data systems in Python
• Deep expertise in PySpark and distributed data processing — Glue, EMR, or Databricks
• Strong experience with lakehouse architectures: Iceberg, Delta Lake, or Hudi on S3
• Production experience with Airflow or a comparable workflow orchestrator
• Solid AWS production experience across S3, Glue, Athena, Lambda, and SQS
• A track record of improving data quality, governance, and pipeline reliability at scale
• You don't need to tick every box — if you're a strong Python data engineer who's excited by lakehouse architecture, streaming pipelines, and working in a genuinely AI-first engineering culture, we'd love to hear from you

Desirable

• If you want to own the data infrastructure that powers real-time pricing decisions for thousands of hotels worldwide — and you're energised by the idea of doing it in an engineering culture where AI isn't a future ambition but how work actually gets done today — this is the role
• You'll be the technical authority on Duetto's data lakehouse, driving everything from pipeline architecture and data quality to the shift from batch to near-real-time streaming
• You'll own the design, performance, and reliability of Duetto's data lakehouse — evolving the Python/PySpark pipeline framework across a bronze → silver → gold architecture on AWS, including Glue jobs, Iceberg MERGE operations, schema evolution, and partitioning strategies
• You'll architect the shift from batch to near-real-time streaming, building SQS-driven stream pipelines with Iceberg sinks and expanding ingestion, normalisation, and analytics layers across the full lakehouse
• You'll drive data quality and governance at scale — extending the Great Expectations framework, leading adoption of data contracts to formalise schemas between producers and consumers, and owning the Athena SQL layer that analysts and product teams depend on
• You'll strengthen observability and reliability through Datadog, Sentry, and Sumo Logic, while optimising Glue job performance — worker sizing, DPU allocation, Spark tuning, and cost management
• You'll build and maintain shared internal Python libraries published to JFrog, and drive improvements to GitHub Actions, Docker-based testing, and CI/CD deployment workflows
• You'll work AI-first every day — using Claude Code and MCP tools in your regular workflow, and contributing to AI-assisted pipeline generation, schema inference, and automated data quality alongside a custom multi-agent system with 17 specialised agents
• Team: Data Platform
• Direct reports: None