Senior Data Engineer, Kitman Labs

• Proven Experience in Data Engineering: A strong track record of designing, building, and optimising data-intensive systems and large-scale ETL/ELT pipelines
• Expertise in the Modern Data Stack: Deep, hands-on experience with cloud-based data platforms, with a strong preference for Google Cloud Platform (GCP). AWS knowledge a plus, but not essential
• Specialised GCP Skillset: Demonstrable, practical experience using GCP Datastream (or similar technology) for Change Data Capture (CDC) and Dataform (or similar tools) for developing and managing data transformations. Proficiency with BigQuery is essential
• Strong Data Modeling Skills: Extensive experience designing and implementing data models (e.g., dimensional modeling, data vault) optimised for analytical workloads and BI tools
• Advanced SQL & Programming: Expertise in advanced SQL for complex data manipulation and analysis, coupled with proficiency in a programming language like Python for automation and scripting
• Performance Tuning & Optimisation: A proven ability to diagnose and resolve performance issues within data pipelines and databases. You understand query optimisation, indexing, and partitioning strategies

Desirable

• We are seeking an experienced and highly skilled Senior Data Engineer to play a pivotal role in the evolution of our analytics platform
• This mission-critical project involves augmenting our in-house platform with cutting-edge data engineering technologies on Google Cloud Platform (GCP) to achieve new levels of scale and performance, complemented by Looker for best-in-class visualization and analysis
• This role will be central to this transformation, working within the team to architect and build the data foundation for our next generation of analytics
• This position is ideal for an engineer who thrives on complex data challenges, including designing robust data models, implementing near real-time data replication using Change Data Capture (CDC), and building highly performant and scalable data transformation pipelines to handle complex business calculations across large datasets (over 300 million data points per customer)
• As a senior team member, you will drive data architecture and best practices, ensuring our new platform is performant, reliable, and capable of delivering the dynamic, insightful reporting our clients depend on
• Driving Data Architecture: Design and build a scalable, end-to-end data architecture on GCP
• This includes creating robust and efficient data models in our data warehouse, defining data flows, and ensuring the infrastructure is optimised for high-volume, near real-time data processing
• Building & Optimising Data Pipelines: Develop, deploy, and manage resilient data pipelines for large-scale data ingestion and transformation
• You will be hands-on with GCP DataStream to implement CDC and orchestrate complex SQL-based transformation workflows with Dataform
• Solving Complex Data Challenges: Tackle and resolve complex performance bottlenecks across the entire data stack
• This involves optimising intricate calculations, tuning database performance, and ensuring the efficiency of our data models to support low-latency queries from Looker
• Upholding Data Quality & Integrity: Champion and implement best practices for data quality, testing, and governance
• You will establish robust data validation checks and build out CI/CD pipelines for all data processes to ensure the accuracy and reliability of our reporting
• Technical Leadership & Mentoring: Provide technical guidance and mentorship to other engineers on data engineering best practices
• You will lead technical decisions, evaluate trade-offs, and foster a culture of data excellence within the squad
• Stakeholder Collaboration: Work in close partnership with product managers and front-end engineers to deeply understand user requirements and translate them into effective data solutions that power our embedded analytics features