Data Engineer, ProblemShared

• Proven experience with a variety of ingestion frameworks (e.g. Fivetran) and analytical databases / lakes (e.g. Redshift, Snowflake, Databricks)
• Production experience working in AWS and infrastructure as code (Terraform, Azure DevOps or GitHub Actions) in an established data team
• Hands-on with Databricks on AWS: jobs, notebooks, IAM integration
• Strong Python and/or R skills, and comfort working with non-relational data formats
• Strong SQL fluency against Postgres and cloud stores
• Action orientation: we value thoughtful, reliable solutions built to deliver immediate value

Desirable

• Work pattern: This is a permanent role, working 40 hours per week Monday to Friday
• Reporting to: Head of Analytics
• About the Job
• ProblemShared has a newly formed Data Office, and we’re looking for a Data Engineer to work closely with our Senior Analytics Engineer, ensuring our data is accessible, accurate and actionable
• We believe in “data for good” - using insights to help neurodivergent people better understand themselves, enable practitioners to deliver more effective care, and guide teams in improving support
• At ProblemShared, we blend people, data and technology to support individuals on their mental health or neurodevelopmental journey - and we believe you can do the most meaningful work of your career here
• We're building a modern data platform (Fivetran → Databricks + DBT → PowerBI) that integrates data from clinical, operational and corporate systems into a safe, accessible, and reliable model, enabling real-time insight and continuous improvement
• You own the “on-ramp” for all our operational data—building and maintaining ingestion pipelines from our AWS-hosted Postgres system, Zendesk, billing feeds, and more into Databricks
• While your primary focus is ingestion, you’ll work hand-in-glove with the Analytics Engineer to understand downstream needs, help troubleshoot transformations, and step in to assist with Silver-layer tasks when peers are out
• You’ll also implement authentication, anonymisation and any non-SQL transformations needed upstream of our analytic models
• Design, deploy and monitor Fivetran (or equivalent) connectors for PostgreSQL, Zendesk, Semble, Bob, and other API and flat-file exports
• Develop Databricks jobs and notebooks to preprocess raw data: validation, pseudonymisation, format conversions (JSON/Parquet)
• Implement and maintain AWS-integrated authentication and role-based access controls in Databricks. (NB. User authentication in ProblemShared uses Microsoft, but our core internal tech stack is on AWS)
• Write scripts (Python or R) to handle edge-case formats and automate testing/alerting
• Collaborate daily with the Analytics Engineer—providing context on schema changes, edge cases and PII nuances
• Cross-train so you can assist with Silver-layer cleanup or Gold-layer hand-offs during leave or as part of some spikes of work
• Partner closely with Software Engineers in our core system teams
• Continuously evaluate and bring in new ingestion or analytical tools (e.g. DuckDB, LLMs) where it adds speed, resilience or adaptability

• Once we review your CV a member of our recruitment team will be in touch to learn more about your experience and motivations
• Technical Interview
• The next step is to meet with our hiring team who will take you through an interview to allow you to demonstrate your ability and suitability for the role. If successful we will move to an offer of employment and pre-employment checks