Staff Data Engineer, Wayve

• Proficiency in Python and SQL, with experience in frameworks like Pandas, PySpark, and NumPy for large-scale data processing
• Expertise in debugging and optimising distributed systems with a focus on scalability and reliability
• Proven ability to design and implement scalable, fault-tolerant ETL pipelines with minimal manual intervention
• Knowledge of data modelling best practices, including the medallion architecture or comparable frameworks
• Experience in workflow orchestration using Flyte, dbt, Airflow, or Prefect
• Strong understanding of unit, integration, and data validation testing using tools like Pytest or Great Expectations
• Familiarity with cloud infrastructure (preferably Azure) for managing pipelines and storage
• Ability to collaborate closely with stakeholders to understand requirements and shape data pipelines to meet user needs effectively

Desirable

• Wayve's machine learning-first approach relies on high-quality, well-structured data
• The Evaluation Workflows and Measurement teams build tools and pipelines that power model evaluation at scale
• As we scale our evaluation approaches and tooling, we need to process massive volumes of test data efficiently and reliably
• This Data Engineer will be embedded in the AI Evaluation division to ensure our evaluation and analytics pipelines are robust, performant, and future-proof
• Their work will strengthen our data foundations for fast decision-making, accelerate the availability of large-scale image and video analytics, and help us rapidly integrate and leverage data from external partners - enabling faster iteration across both offline and on-road evaluation
• Build scalable and reliable data and analytics pipelines to process and enrich over 1 million hours of driving video data annually and supply mission-critical data to stakeholders across the business
• Unlock rapid insights by architecting and optimising analytics pipelines that drive company wide development and decision-making
• Collaborate across functions - including research engineers, simulation experts, robotics engineers, data scientists and safety drivers - to deliver and visualise enriched data
• Improve pipeline observability, validation, and fault tolerance for production-grade robustness
• Enable LLM-driven workflows by shaping data to be AI-consumable (e.g. chunking, embeddings, metadata)
• Reduce tech debt and simplify orchestration across Flyte, Databricks, and Azure-based infrastructure
• Example Projects:
• Design and optimise distributed data pipelines to handle large-scale video and image data processing
• Re-design and optimise existing analytics pipelines
• Collaborate with the data platform team to integrate pipelines with Databricks for governance and compliance - and unlock massive scale for offline evaluation from third party datasets
• Shape evaluation data to support future use cases like Retrieval-Augmented Generation (RAG) and natural language analytics