Senior Data Engineer, ProblemShared

• Proven experience with a variety of ingestion frameworks (e.g. Fivetran) and analytical databases / lakes (e.g. Redshift, Snowflake, Databricks)
• Production experience working in AWS and infrastructure as code (Terraform, Azure DevOps or GitHub Actions) in an established data team
• Hands-on with Databricks on AWS: jobs, notebooks, IAM integration
• Strong Python and/or R skills, and comfort working with non-relational data formats
• Strong SQL fluency against Postgres and cloud stores
• Action orientation: we value thoughtful, reliable solutions built to deliver immediate value

Desirable

• This is a permanent role, working 40 hours per week Monday to Friday
• You own the “on-ramp” for all our operational data—building and maintaining ingestion pipelines from our AWS-hosted Postgres system, Zendesk, billing feeds, and more into Databricks
• While your primary focus is ingestion, you’ll work hand-in-glove with the Analytics Engineer to understand downstream needs, help troubleshoot transformations, and step in to assist with Silver-layer tasks when peers are out
• You’ll also implement authentication, anonymisation and any non-SQL transformations needed upstream of our analytic models
• Design, deploy and monitor Fivetran (or equivalent) connectors for PostgreSQL, Zendesk, Semble, Bob, and other API and flat-file exports
• Develop Databricks jobs and notebooks to preprocess raw data: validation, pseudonymisation, format conversions (JSON/Parquet)
• Implement and maintain AWS-integrated authentication and role-based access controls in Databricks. (NB. User authentication in ProblemShared uses Microsoft, but our core internal tech stack is on AWS)
• Write scripts (Python or R) to handle edge-case formats and automate testing/alerting
• Collaborate daily with the Analytics Engineer—providing context on schema changes, edge cases and PII nuances
• Cross-train so you can assist with Silver-layer cleanup or Gold-layer hand-offs during leave or as part of some spikes of work
• Partner closely with Software Engineers in our core system teams
• Continuously evaluate and bring in new ingestion or analytical tools (e.g. DuckDB, LLMs) where it adds speed, resilience or adaptability

• 1) Screening
• Once we review your CV a member of our recruitment team will be in touch to learn more about your experience and motivations
• 2) Technical Interview
• The next step is to meet with our hiring team who will take you through an interview to allow you to demonstrate your ability and suitability for the role. If successful we will move to an offer of employment and pre-employment checks
• We know that applying for a new role and considering a new team to join can be a daunting process. If you have never worked remotely before, this can also generate a lot of questions regarding the environment and ways of working. We aim to make your experience with us as informative and enjoyable as possible