Data Engineer, Trustpair

• 3-5 years of experience as a Data Engineer
• Ability to maintain and improve a whole Data Lake and/or Data Warehouse
• Good knowledge on data lake architectures and ingestion patterns
• Understanding on building a production system with performance and reliability drivers
• Professional experience with Python
• Good knowledge of Cloud environments (ideally AWS)
• Experience in Spark, Airflow or equivalents
• Good understanding of SQL
• Fluent in English
• Strong problem solving skills
• Ability to work autonomously

Desirable

• As the referent Data Engineer at Trustpair, reporting to Cyrille, our VP Operations and working closely with our SRE, VP Engineering and CTO, you will be responsible for leading a significant improvement project (reliability, automation, infra, security)
• Working on on our existing AWS Data lake, you'll help our stack scale and answer our current and future Data Analytics needs, as well as evolving product-related Data needs
• This is a freelance contract, working five days per week
• The improvement project will last at least 2 months, and may be extended
• Upon completion there is the possibility to adopt a 2 days per week contract to maintain the stack
• Our current stack on AWS is: S3 (Lake), DMS (data change capture), Airflow (Orchestrator), DBT, Athena (query engine), Glue, EMR
• Initial audit:
• Assess our technical stack with the current use cases in mind and challenge our patterns
• Document the processes, the guidelines and when to use Airflow / DMS / DBT
• Address the monitoring and observability of the stack: monitoring / alerting
• Deploy some data drift control mechanisms and realignment processes
• Challenge the staging environment to make it more meaningful before going to prod
• For now we have one Data Ingestion pattern by assessing the postgresql wal file and reconstructing images
• Deploy a new Ingestion pattern that could easily integrate within the data lake
• Migrate Airflow to Kubernetes with our SRE
• Adapt Airflow configuration to support Google Authenticator
• Challenge our EMR setup
• Challenge Data security (RBAC) within the data lake
• Stack day to day maintenance
• Ingestion of new sources when required, to support our analytics projects

• First call with Aicha, Talent Acquisition (30 min)
• Experience interview with Cyrille, VP Operations (30 min)
• Technical test (at home) & interview with our SRE and VP Engineering (1h)
• Coffee fit with two members of Trustpair & one of our Co-founders (1h)