Data Engineer, Homeprotect

• Python - Candidates should have strong Python programming skills, including a solid understanding of object-oriented programming (OOP), asynchronous programming, and containerisation (e.g., Docker). Experience in developing and deploying production-grade Python applications, solving complex problems, and collaborating on projects with innovative technical solutions is essential
• SQL – Candidates should have strong SQL skills, including the ability to write and optimise complex queries, design and manage database structures, as well as create and debug stored procedures. Experience in building ETL/ELT pipelines, performing data transformation, and optimising performance is essential, along with a good understanding of data integration and collaboration with cross-functional teams
• Terraform – Candidates should have practical experience with Infrastructure as Code using Terraform, including designing, deploying, and managing cloud infrastructure. Proficiency in writing and managing Terraform modules, following best practices for code structure and version control, and ensuring scalable, secure infrastructure is essential. Familiarity with Azure as a cloud provider is preferred
• Cloud data and analytics platforms – Candidates should have experience working with cloud platforms, with a strong preference for experience in Azure
• Data Understanding – Candidates should have a strong understanding of different types of data, including structured/unstructured data and streaming/batch data, and how to handle them effectively. This includes knowledge of data organisation and storage formats, with experience using distributed systems such as Azure Data Lake Storage or similar solutions for managing both types of data. Familiarity with unstructured data types, such as images, PDFs, videos, and other media, as well as an understanding of their applications in machine learning and AI projects, is highly desirable

Desirable

• Design, develop, and maintain customised data systems for collecting, processing, and provisioning data from internal and external sources, ensuring data consistency and quality to meet business needs
• Develop and maintain Terraform modules for new and existing data architecture components, ensuring code is modular, well-documented, and adheres to best practices for scalability, maintainability, and security
• Collaborate closely with the DevOps team to align data engineering requirements with deployment strategies and CI/CD pipeline integration. Represent the data engineering team in cross-functional meetings, advocating for its needs and ensuring that infrastructure changes are well-coordinated and communicated
• Assist the team with integrating developed solutions into existing processes. This includes adding new pipelines to Airflow and DBT, updating configurations in current solutions, and collaborating with DevOps to set up infrastructure and address integration-related questions. Take ownership of the deployment and integration phases of projects, ensuring thorough testing, automation, and monitoring to maintain stability. Document processes clearly to support team continuity and allow colleagues to focus on enhancing solution functionality
• Collaborate with data scientists, analysts, software developers, and other stakeholders, taking ownership of delivering data products as needed for each project