Data Engineer, Zeki

• Have a BSc in Computer Science or related field (Mathematics, Physics, Economics etc.)
• 3-5 years of experience as data engineer working in AWS infrastructure.
• Strong engineering capability in data security, data management, DevOps, data architecture, orchestration, and software engineering.
• Have a good understanding of complex datasets and excited to gain domain knowledge in the field of opensource research data
• Critical thinking around complex problems is a must while providing effective solutions which can be implemented in a timely manner
• AWS Services: Athena, S3, EC2, Glue Catalogue, Glue Workflows, RDS
• Strong Coding experience: Expert in SQL and python. Experience using terraform and spark. Beneficial to have worked in other languages such as JavaScript, R etc
• You are passionate about incubating new ideas, solving problems, and building working systems
• You are self-motivated, proven collaborators, good communicators, attentive to details, and keen to learn.
• SQL knowledge should extend from the basics all the way through to window functions, crossjoins, coalesce, views, materialised views and optimising slow query performance
• Strong python skills especially in frameworks such as Django, DBT and when working with APIs and databases
• Orchestration: Experience using some form of orchestration: AWS Glue workflows, Airflow, Celery, CRON jobs etc
• Storage: A strong understanding of S3 from lifecycle management, permissions and working with it through programmatic access
• Databases: Experience using Postgres or a similar engine while having designed schemas from scratch. Athena/Presto with S3 as the storage layer
• DataOps: deploying pipelines to AWS using CI/CD or IaC tools while keeping to software engineering best practices in code commits
• Engineering: Experienced in using version-controlled systems such as GitHub with regular code commits, pull requests, branching and merging. This will extend to running github pipelines to automate deployments through CI/CD yaml files
• Solid experience with big data technologies such as presto, spark and their underlying storage formats (parquet, jsonl). Must have knowledge of compression formats such as snappy/gzip/orc, while knowing when to use partitioning, bucketing and sorting on large files to optimise performance
• Additionally, experience with infrastructure as code tools such as terraform is beneficial
• Must legally be able to work in the UK and, where required, hold a valid visa to support your Right to Work.

Desirable

• Design, implement, and maintain data processing pipelines to aggregate, clean, and process large volumes of data
• Develop and optimise data models, algorithms, and indexing techniques to enable efficient querying and analysis of data
• Collaborate with cross-functional teams to understand their data needs. Architect solutions to data challenges
• Ensure data quality, integrity, and security by implementing appropriate data validation, storage, and access controls
• Monitor and troubleshoot data processing systems to maintain optimal performance, identify areas for improvement, and recommend enhancements
• Document data processing procedures, architecture, and best practices to foster knowledge sharing within the organisation
• Stay up-to-date with the latest trends, tools, and technologies and data processing domains to drive innovation and continuous improvement

• 1. Apply via Otta.
• 2. Team's video call with a member of Zeki's HR recruitment Team.
• 3. Team's video call with the Head of Data.
• 4. Either in person or a Teams video call with the CEO and Head of Data.