Data Engineer, Curve

• This role is a great opportunity for people who love data, software architecture, streaming technologies, distributed systems and who want to have a real-world impact by building production systems from scratch with a world class team
• At least 3+ years’ experience as a data engineer especially using realtime & batch data in a production environment using streaming technologies such as Kafka/PubSub/Kinesis
• Understanding of Data Warehousing principles (including indexing, query graphs, basic administration, relational models, etc)
• You have a proven track record with a programming language. We are especially keen on any or all of the following Python, Go and SQL, specifically in connection with Airflow and DBT respectively
• Experience designing and building production systems on GCP and/or AWS infrastructure
• Understanding and/or experience of working within Event Driven Architectures
• Experience deploying and maintaining both batch and real-time machine learning models
• Experience using distributed systems at scale in a production environment using technologies such as Spark/Beam
• Strong understanding of data quality principles
• A record of learning new technologies and tools
• You are experienced with database technologies. (Best Practice, Performance Optimisation, Fault Finding)

Desirable

• We’re looking for a capable Senior Data Engineer, who will be part of the data platform team
• Our mission is to build a robust platform to collect data from multiple data sources, both internal and external as well as enable reverse-ETL
• Using a combination of batch and real-time streaming technologies, the role will facilitate cutting-edge analytics, real-time decisioning, and robust reporting
• We believe in data being at the core of the business. We hate latency, approximation and bad data quality!
• Managing and administering our Data Warehouse to guarantee continuous and high availability of the data
• You will mainly be working with GCP (Cloud Storage, BigQuery, Composer, Dataproc, Dataflow), Confluent for Kafka and Snowplow
• Working with the extended team to understand their needs and translating them into fantastic data products, this includes working with and alongside Machine Learning researchers and data scientists on cutting edge ML models
• Designing and building pipelines to collect data from various data sources
• Data modelling using DBT on top of BigQuery
• Working with backend, frontend, Devops & QA engineers to ensure that data events are well-designed and correctly integrated to data pipeline services
• Helping us implement a data-driven mindset in the company