Senior Data Engineer, Vitesse

• The successful person will have experience building complex data frameworks in other early-stage businesses
• They will have experience with cloud platforms, ideally AWS
• Applicants with the following technology experience will be preferred:
• AWS
• Airflow
• Databricks
• GitHub
• Great Expectations
• PyTest
• Kafka
• Mosaic AI
• Terraform
• Tableau
• Experience with data pipeline orchestration tools such as Airflow, Luigi, or similar
• Experience with version control systems and CI/CD best practices using GitHub Actions
• Knowledge of data governance, privacy regulations (e.g., GDPR), and security best practices
• Proficiency with SQL and experience with distributed data processing tools such as Apache Spark
• Strong understanding of relational and NoSQL databases (e.g., PostgreSQL, MongoDB, Impala, Cassandra)
• Experience with cloud infrastructure (Docker and Kubernetes, Terraform)
• Experience in AWS platform architecture and cloud services
• A collaborative team member with Agile experience
• Familiarity with stream processing technologies either Kafka or Kinesis

Desirable

• Your role will be to help architect and build a robust, secure and scalable data architecture to support real-time observability and reporting
• This will entail the design and construction of data engineering solutions on AWS, as well as the design and construction of data service API's
• You will be supporting key internal and external stakeholders that require access to real-time and curated data
• In this role, you will be working closely with internal stakeholders across the organisation, rapidly becoming a subject matter expert of the data generated by Vitesse and our ELT pipelines as well as drive data architecture strategy and Terraform deployment
• Design, build, and maintain scalable data pipelines and architectures to handle large volumes of structured and unstructured data
• Develop, enhance, and optimize ELT processes for ingesting, processing, and distributing data across multiple platforms in real time
• Build and manage data warehouses (to support advanced analytics, reporting, and machine learning
• Implement data governance, quality checks, and validation processes to ensure the accuracy, consistency, observability, and security of data
• Optimize query performance and data storage costs through techniques like partitioning, indexing, vacuuming, and compression
• Build monitoring and alerting systems for data pipelines to proactively detect and resolve issues before they impact business operations
• Optimize existing data pipelines for better performance, cost-efficiency, and scalability
• Work with data scientists, analysts, and business stakeholders to understand data needs and ensure availability of relevant data
• Continuously research and integrate cutting-edge data technologies, tools, and practices to improve data engineering processes
• Team up with product engineers to identify, root cause, and resolve bugs
• Update documentation to help users navigate our data products
• Ensure our data platform performs well and is always available for our blue-chip clients