Senior Data Engineer, Leadtech

• A successful candidate has extensive knowledge of cloud-native data solutions, strong proficiency with ETL/ELT frameworks (including dbt), and a passion for building robust, cost-effective pipelines
• 3+ years of professional experience in data engineering, with at least 1 year in mobile data
• Proven track record building and maintaining BigQuery environments and Google Cloud Storage based data lakes
• Deep knowledge of Apache Airflow for scheduling/orchestration and ETL/ELT design
• Experience implementing dbt for data transformations, RabbitMQ for event-driven workflows, and Pub/Sub + Dataflow for streaming/batch data pipelines
• Familiarity with designing and implementing Data Mart (Data Mach) solutions, as well as using Terraform for IaC
• Strong coding capabilities in Python, Java, or Scala, plus scripting for automation
• Experience with Docker and Kubernetes (K8s) for containerizing data-related services
• Hands-on with CI/CD pipelines and DevOps tools (e.g., Terraform, Ansible, Jenkins, GitLab CI) to manage infrastructure and deployments
• Proficiency in Great Expectations (or similar) to define and enforce data quality standards
• Expertise in designing systems for data lineage, metadata management, and compliance (GDPR, CCPA)
• Strong understanding of OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing) systems
• Excellent communication skills for both technical and non-technical audiences
• High level of organization, self-motivation, and problem-solving aptitude

Desirable

• We are looking for a Senior Data Engineer to design, develop, and optimize our data infrastructure on Databricks
• You will architect scalable pipelines using BigQuery, Google Cloud Storage, Apache Airflow, dbt, Dataflow, and Pub/Sub, ensuring high availability and performance across our ETL/ELT processes
• You will leverage Great Expectations to enforce data quality standards
• The role also involves building our Data Mart (Data Mach) environment and implementing CI/CD best practices
• Define and implement the overall data architecture on GCP, including data warehousing in BigQuery/Databricks, data lake patterns in Google Cloud Storage, and Data Mart (Data Mach) solutions
• Integrate Terraform for Infrastructure as Code to provision and manage cloud resources efficiently
• Establish both batch and real-time data processing frameworks to ensure reliability, scalability, and cost efficiency
• Design, build, and optimize ETL/ELT pipelines using Apache Airflow for workflow orchestration
• Implement dbt (Data Build Tool) transformations to maintain version-controlled data models in BigQuery, ensuring consistency and reliability across the data pipeline
• Use Google Dataflow (based on Apache Beam) and Pub/Sub for large-scale streaming/batch data processing and ingestion
• Automate job scheduling and data transformations to deliver timely insights for analytics, machine learning, and reporting
• Implement event-driven or asynchronous data workflows between microservices
• Employ Docker and Kubernetes (K8s) for containerization and orchestration, enabling flexible and efficient microservices-based data workflows
• Implement CI/CD pipelines for streamlined development, testing, and deployment of data engineering components
• Enforce data quality standards using Great Expectations or similar frameworks, defining and validating expectations for critical datasets
• Define and uphold metadata management, data lineage, and auditing standards to ensure trustworthy datasets
• Implement security best practices, including encryption at rest and in transit, Identity and Access Management (IAM), and compliance with GDPR or CCPA where applicable
• Collaborate with Data Science, Analytics, and Product teams to ensure the data infrastructure supports advanced analytics, including machine learning initiatives
• Maintain Data Mart (Data Mach) environments that cater to specific business domains, optimizing access and performance for key stakeholders