Senior Software Engineer, Sentry
AI Evals
$240-280k
+ Equity
More information about location
Error tracking and app monitoring software
Open for applications
Error tracking and app monitoring software
201-500 employees
Open for applications
$240-280k
+ Equity
More information about location
201-500 employees
Company mission
To empower developer teams to build their best possible software, more quickly and easily.
Role
Who you are
- Care deeply about correctness, rigor, and measurement in AI systems
- Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics
- Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team
- Thrive in cross-functional environments and enjoy influencing model design through better evaluation
- Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
- Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
- Comfort writing production-quality code (we use Python and TypeScript)
- Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
- Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
- Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools
What the job involves
- As a Senior Software Engineer on Sentry’s AI/ML team, you’ll be responsible for building the evaluation infrastructure that measures the accuracy, reliability, and real-world performance of our AI systems
- This role is critical to ensuring that our debugging agents and AI-powered features behave correctly, safely, and predictably as they scale
- You’ll design datasets, benchmarks, and test harnesses that turn ambiguous AI behavior into measurable signals, helping the team ship AI with confidence
- Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
- Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
- Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
- Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
- Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring
Our take
Defective software leads to dissatisfied users, disruptions and potential financial losses for companies. Sentry has created application monitoring and error tracking software to help developers fix errors faster, easier and more cost effectively.
More than 1 million developers and 70,000 organisations worldwide, including the likes of Microsoft, Disney, Atlassian, Cloudflare and Slack, have used Sentry to monitor their code health. The company is well-positioned for significant growth. There is an emerging need for solutions that address holistic application health. As more companies focus on digital to engage with their customers, the reliability of their applications is essential.
The global bug-tracking software market is expected to reach over $600 million by 2026 with an annual growth rate of almost 10%. Further, the company opened another office in Denver in early 2023, expanding its regional presence, as it continues to broaden its market presence.
Freddie
Company Specialist at Welcome to the Jungle
Insights
28% employee growth in 12 months
Company
Funding (last 2 of 6 rounds)
May 2022
$90m
SERIES E
Feb 2021
$60m
SERIES D
Total funding: $216.5m
Company benefits
- Competitive Compensation + Equity
- 401k Plan
- Medical, Dental, Vision Insurance
- Commuter Stipend
- Professional Development Stipend
- Health & Wellness Benefits
- Charitable Matching Program
- Flexible PTO
- Paid Parental Leave
Company HQ
Financial District, San Francisco, CA
Leadership
David Cramer
(Co-Founder & CPO)They started their career as a Software Engineer at Curse, before roles at Enthropia, Disqus, and Dropbox. Having co-founded Sentry in January 2012, they left Dropbox in 2015.
Chris Jennings
(Co-Founder & Chief Creative Officer)They have been a Frontend Developer at AgencyTen, Director of Design and UX at Disqus and most recently a Product Designer at GitHub.
Salary benchmarks
We don't have enough data yet to provide salary benchmarks for this role.
Submit your salary to help other candidates with crowdsourced salary estimates.
Share this job
View 28 more jobs at Sentry