Senior Data Engineer, GitGuardian

• 5+ years of experience in data engineering, building and running production pipelines on large volumes (billions of rows)
• Solid data modeling for pipelines: you choose the grain, keys, deduplication and storage of each dataset from how it will be consumed, and you can defend these choices
• Expert Python and SQL
• Hands-on experience with a large-scale analytical database (ClickHouse, Snowflake, BigQuery or similar) and with data on object storage (S3, Parquet)
• A good understanding of data-intensive services: queues, workers, retries, deduplication, rate limits. You don't need to be a backend expert, but you can follow and challenge a design discussion about them
• A production mindset: you monitor what you ship, check that your data is complete, and take your share of the run
• Deliberate use of AI coding agents: you give them context and a spec, read what they produce, and catch what they get wrong
• Clear communication: you explain your choices to engineers, product and sales, and you hold your position when you have good reasons to
• Fluent English in an international environment
• Building data for LLM agents: context retrieval, evaluation datasets

Desirable

• You will join the Public Intelligence squad, which builds the engine behind GitGuardian's mission to protect companies from secrets leaked in public
• We collect public data at scale, from every public commit pushed to GitHub to public Docker Hub images, and find the secrets in it
• AI agents then work out which company each leak belongs to and how critical it is
• Customers see these leaks in their GitGuardian workspace, and the most critical ones help our sales team start conversations with new prospects
• Data is at the heart of the squad. We ingest about 12.5 million public GitHub commits a day and store nearly 400 million secret occurrences in ClickHouse
• The systems run in production, and AI agents already analyze every public secret we find
• They grew fast, one project at a time, and we now have a real job ahead of us: rethink how we collect, store and process public data, so the platform is faster, cheaper, more reliable and ready for new sources. That is the work this role is for
• You will join a squad of 5 engineers (software, data and AI) and a Product Manager, and report to Valentin, the squad's Engineering Manager
• Build a single source of truth for public data. Move public data and its transformations out of Snowflake into an S3 + Iceberg data lake that every GitGuardian service can rely on
• Never miss a public leak. The GitHub public feed is huge, rate-limited and noisy. Collection must keep up with it, measure how much of it we cover, and recover quickly after an incident
• Go from hours to minutes. Some agent analyses still wait for a daily batch in Snowflake. A leak is only actionable while it is fresh, so the goal is a qualified incident within minutes of the commit landing on GitHub
• Scale historical scans. When a company starts monitoring, we scan the public history of its developers across billions of commits. Moving secret attribution to ClickHouse will make these scans faster and cheaper
• Industrialize new sources. Make the collection of new sources as robust as GitHub's, and simplify their integration so each new source is cheaper to add than the last
• Design, build and run the squad's data pipelines, from collection to the tables our agents, our product and our sales team consume
• Model public data: choose the grain, keys and storage (S3, ClickHouse, Snowflake, PostgreSQL) according to what each consumer needs
• Orchestrate pipelines with Dagster and deploy them with Docker, Kubernetes and Terraform
• Make coverage and data quality measurable, with monitoring and alerts that catch silent data loss before a customer does
• Take part in the run: investigate data issues and make sure every critical alert comes with a runbook
• Work closely with the squad's software engineers, who build the collectors and the product, and its AI engineers, who build the agents on top of your data
• Help build the squad's software factory. AI agents write a growing share of our code, and you will shape how they work: write the skills, tools and conventions that make them reliable, so the whole squad ships faster without lowering the bar
• Technical environment:
• Languages: Python, SQL
• Orchestration: Dagster, Celery, AWS Batch
• Storage and analytics: S3, Iceberg, ClickHouse, Snowflake, PostgreSQL, Redis
• Messaging and ingestion: RabbitMQ, Vector
• Deployment: Docker, Kubernetes, Terraform, ArgoCD, GitLab CI
• Cloud: AWS, OVH
• AI: LangGraph, LiteLLM, Claude Code
• Observability: Coralogix, Sentry, OpenTelemetry

• Video call with a recruiter:
• To discover your professional project and evaluate if there could be a mutual match
• Interview with your future manager (1h):
• Purpose: To know more about you and your achievements, and to present the squad
• Skills Assessed: Ownership, communication, motivation, and how you use AI in your work
• Technical interview with the team (2h):
• Purpose: A system design discussion and a hands-on data modeling exercise, both on public GitHub data
• Skills Assessed: Data modeling, pipeline design and understanding of ingestion systems
• Final interviews with Pierre, VP of Engineering, and Eric, Co-founder and CEO:
• Purpose: To know more about your ambitions, and to share our company's vision for the next couple of years
• References check:
• You can start thinking about two contacts who can attest to your previous or current professional experiences. These contacts should be as recent as possible, and we will call them at the end of the process