AI Evaluation Engineer, Distyl
$150-250k
Plus equity
More information about location
Generative AI solutions for enterprises
Be an early applicant
Generative AI solutions for enterprises
101-200 employees
Be an early applicant
$150-250k
Plus equity
More information about location
101-200 employees
Company mission
To create the most customer-centric AI company that revolutionizes how enterprise thrive in the AI-assisted economy.
Role
Who you are
- AI Evaluation Engineers focus on designing and implementing the evaluation systems that drive this process
- They are hands-on engineers who write production Python code, build evaluation pipelines, and use structured signals to guide system design, prompt iteration, and deployment decisions for real customer-facing AI systems
- This role is for engineers who believe that AI systems only improve when measurement is tightly coupled to development—and who want to apply that philosophy directly to systems that matter
- 2+ years of software engineering experience
- Strong Python Engineering Skills: Write clean, maintainable Python and are comfortable building evaluation and experimentation pipelines that run in production environments. You treat evaluation code with the same rigor as application code
- Experience with Evaluation-Driven or Experiment-Driven Development: Experience using structured evaluation or experimentation frameworks to drive system iteration, and understand the pitfalls of overfitting to metrics that don’t reflect real outcomes
- Ability to Translate Human Judgment into Code: Work with subject matter experts to elicit high-quality judgments and encode them into test cases, scoring functions, and graders that scale
- Systems-Oriented Mindset: Understand how evaluation interacts with prompts, agents, data, and deployment. You design evaluation systems that support fast iteration while maintaining trust and safety in production
- AI-Native Working Style: Use AI tools to generate tests, analyze failures, explore edge cases, and accelerate debugging and iteration
- Travel: Ability to travel 25-50%
What the job involves
- At Distyl, we build AI systems using Evaluation-Driven Development—an approach where evaluation is not an afterthought, but the primary mechanism for iterating, improving, and trusting AI behavior in production
- Design and implement evaluation frameworks that enable Evaluation-Driven Development for AI systems deployed in customer environments
- Define how system quality is measured in each domain, ensuring that evaluation signals reflect real user needs, domain constraints, and business objectives
- Build and maintain golden test cases and regression suites in Python, using both human-authored and AI-assisted test generation to capture critical behaviors and edge cases. These test suites are treated as first-class system components that evolve alongside the AI system itself
- Develop and maintain evaluation pipelines—offline and online—that integrate directly into system iteration loops. Evaluation results inform prompt design, agent logic, model selection, and release readiness, ensuring that system changes are driven by measurable improvements rather than intuition alone
- Define, calibrate, and operate LLM-based graders, aligning automated judgments with expert human assessments. They investigate where evaluation signals diverge from real-world outcomes and refine grading approaches to maintain signal quality as systems and domains evolve
- Work closely with Forward Deployed AI Engineers, Architects, Product Engineers, AI Strategists, and domain experts to ensure evaluation frameworks meaningfully guide system development and deployment in production
Application process
- Due to the high volume of interest, we’re not able to review or respond to all direct emails or LinkedIn messages. We will be in touch with every applicant once we’ve completed our review, regardless of the decision
Our take
Distyl helps large enterprises adopt AI by providing tailored AI solutions. Its platform combines AI-native software with a top engineering team, enabling rapid implementation of AI across industries like healthcare, telecom and retail.
The company uses precision-engineered software and tools like Distillery to automate operations, transforming knowledge into AI processes. This results in faster decision-making and improved efficiency, allowing enterprises to stay competitive and optimise their operations.
With impressive funding secured, Distyl has quickly gained traction from Fortune 100 companies, delivering stunning outcomes like a 47% reduction in task resolution time. Having broken into Unicorn territory with a $1.7 billion valuation within two years of founding, the company is growing and expanding its customer base and product capabilities.
Freddie
Company Specialist at Welcome to the Jungle
Company
Funding (last 2 of 3 rounds)
Sep 2025
$175m
LATE VC
Nov 2024
$20m
SERIES A
Total funding: $202m
Company benefits
- Equity options and healthcare
- Collaborative and supportive work environment
- Strong focus on personal and professional growth
Company HQ
SoMa, San Francisco, CA
Leadership
Arjun Prakash
(Co-Founder & CEO)Previously the Head of Solutions at Snorkel AI, a Business Development Head at Palantir Technologies and an Associate at BlackRock.
Derek Ho
(Co-Founder & COO)Previously the VP of Customers at Mendel.ai and also worked in Business Development at Palantir Technologies.
Salary benchmarks
We don't have enough data yet to provide salary benchmarks for this role.
Submit your salary to help other candidates with crowdsourced salary estimates.
Share this job
View 17 more jobs at Distyl