Staff Data Engineer, H1
Emerald
$170-190k
+ Stock Options
Platform connecting healthcare professionals
Job no longer available
Platform connecting healthcare professionals
501-1000 employees
Job no longer available
$170-190k
+ Stock Options
501-1000 employees
Company mission
To create a healthier future by delivering a platform that connects stakeholders across the healthcare ecosystem and encourages purposeful relationships.
Role
Who you are
- You are an experienced data engineer with deep expertise building and optimizing distributed data systems in cloud-native environments. You thrive solving complex scalability and performance challenges across high-volume data processing systems and enjoy operating in highly technical, fast-paced engineering environments
- You bring strong hands-on engineering expertise across distributed computing, large-scale data processing, and infrastructure optimization while also helping guide technical direction and mentor engineers across the organization
- Deep expertise with distributed data processing frameworks such as Apache Spark and Hadoop, particularly within AWS environments
- Strong proficiency in Python (PySpark), Scala, Java, or other modern programming languages used for large-scale distributed processing
- Experience building scalable ETL/ELT frameworks across both batch and streaming architectures
- Experience with entity resolution, identity mapping, automatching, deduplication, or large-scale matching systems is strongly preferred
- Strong understanding of distributed file formats including Apache Parquet and Apache AVRO
- Experience with streaming technologies such as Kafka, Spark Streaming, or KSQL
- Strong grasp of software engineering fundamentals including distributed systems, data structures, concurrency, and system design
- Experience performing root cause analysis across large-scale distributed systems and complex data pipelines
- Ability to write clean, maintainable, modular, and production-grade code
- Experience improving performance, scalability, observability, and infrastructure efficiency within distributed systems
- Strong communication and collaboration skills across both technical and non-technical stakeholders
- Familiarity with modern development and infrastructure tooling including Git, CI/CD pipelines, Docker, Kubernetes, Terraform, Argo, Hudi, and JIRA
- 8+ years of experience building and maintaining large-scale distributed data systems and pipelines
- Demonstrated technical leadership experience mentoring engineers and driving complex technical initiatives
- Extensive experience with Apache Spark and AWS-based big data technologies including EMR, S3, and distributed compute environments
- Strong coding experience in Python (PySpark), Scala, Java, or equivalent languages used for distributed processing systems
- Experience optimizing large-scale Spark workloads for performance, scalability, and infrastructure cost efficiency
- Experience with streaming and event-driven architectures using technologies such as Kafka or Spark Streaming
- Experience with orchestration and lakehouse technologies such as Argo and Hudi or comparable platforms
- Experience with containerization and infrastructure technologies such as Docker, Kubernetes, and Terraform
- Experience working with relational or distributed databases such as PostgreSQL or Redshift
- Proven ability to operate effectively within highly scalable, production-grade distributed systems
- Experience working with healthcare, life sciences, Real World Evidence (RWE), or large-scale healthcare datasets is strongly preferred
What the job involves
- As a Staff Data Engineer on the Emerald team, you will play a critical role in shaping the architecture, scalability, and technical direction of H1’s healthcare entity resolution platform. EMERALD is responsible for linking large-scale external healthcare datasets, including PubMed, clinical trials, conferences, ct.gov, and web-collected data to H1’s canonical physician and organization profiles
- This role sits at the intersection of distributed data engineering, entity matching, identity resolution, and large-scale healthcare data processing. You will lead a small team of engineers while remaining deeply hands-on technically, owning the systems and pipelines powering automatching, grouping logic, identity mapping, deduplication, and enrichment workflows processing tens of millions of records
- You will partner closely with Product, AI/ML, Analytics, and Engineering teams to improve platform accuracy, scalability, reliability, and operational efficiency across one of H1’s most critical data platforms
- Lead the design, optimization, and scalability of distributed Spark/PySpark pipelines powering entity resolution and large-scale healthcare data processing
- Own systems supporting automatching, identity mapping, grouping logic, deduplication, enrichment, and auto-approval workflows across healthcare provider and organization datasets
- Build and maintain scalable processing frameworks for PubMed, clinical trial, ct.gov, conference, and other healthcare data sources
- Drive infrastructure optimization initiatives focused on improving throughput, runtime, observability, and cloud compute cost efficiency
- Partner closely with AI/ML teams to integrate matching and resolution models into EMERALD and improve matching precision and recall
- Lead complex technical initiatives from architecture and design through deployment, monitoring, and long-term production support
- Serve as a technical leader and mentor across the team through code reviews, technical guidance, and engineering best practices
- Collaborate directly with Product and business stakeholders to align technical solutions with operational and customer needs
- Support production operations, incident response, troubleshooting, and ongoing platform reliability
Application process
- Anticipated role close date: 8/1/2026
Our take
The increasing costs and difficulty of drug discovery requires healthcare organizations and pharmaceutical companies to be able to quickly connect to professionals and thought leaders with specific credentials or research history. H1 addresses these needs by allowing life sciences companies, hospitals and healthcare systems to locate the professionals they need.
H1 is the world’s largest healthcare database. It acts a bit like LinkedIn for healthcare professionals, and has already built a network of more than 9 million users. It has also built a corporate client base of 250 enterprises which is able to access its data which spans more than 250 million patients and over 10 billion health care claims.
The company graduated from Y Combinator's accelerator program in January 2020, and has since raised considerable funding, expanded into Europe and Asia, and built out its data network. It continues to attract investment, most lately from CVS Health Ventures, which showcases the industry regard for its offering.
Freddie
Company Specialist at Welcome to the Jungle
Insights
-7% employee growth in 12 months
Company
Funding (last 2 of 6 rounds)
May 2026
$40m
LATE VC
Jun 2022
$23m
SERIES C
Total funding: $227.2m
Company benefits
- Flexible work hours
- Commuter benefits
- Stock options
- Computer setup
- Work from home opportunities
- Health & life insurance
- Retirement options
- Unlimited PTO
- Flex Give & Flex Spend
- Impactful BRGs
Company HQ
Flatiron District, New York, NY
Leadership
Ariel Katz
(CEO & Co-Founder)Previously founded acquired online database, ResearchConnection, before co-founding H1.
Ian Sax
(Co-Founder)Ian has been building businesses for over two decades. He began his start up career at The Carson Group, where he built the firm's global people, institution, and shareholder platform.
Salary benchmarks
We don't have enough data yet to provide salary benchmarks for this role.
Submit your salary to help other candidates with crowdsourced salary estimates.
Share this job
View 12 more jobs at H1