AI Evaluation Engineer, Dialpad
CA$115.5-132.8k
AI-powered cloud communications platform
Open for applications
AI-powered cloud communications platform
1001+ employees
Open for applications
CA$115.5-132.8k
1001+ employees
Company mission
To completely transform the way the world works together.
Role
Who you are
- Bachelor's or Master's degree in Computer Science, Software Engineering, Computational Linguistics, or a related field
- 3+ years of experience in QA, test engineering, model evaluation, or applied ML quality for AI-driven products
- Experience designing structured test strategies across manual and automated workflows
- Comfort working with complex AI systems such as speech, NLP, LLM, or agentic products
- Experience working with evaluation datasets, gold sets, adversarial test sets, or benchmark creation for AI systems
- Strong analytical skills for investigating failures, comparing outputs, and identifying actionable quality patterns
- Experience collaborating with cross-functional technical teams and communicating clearly through documentation and reporting
- Don’t meet every single requirement? If you’re excited about this role and possess the fundamental traits, drive, and strong ambition we seek, but your experience doesn’t meet every qualification, we encourage you to apply
What the job involves
- As an AI Evaluation Engineer, you'll be an integral part of our AI Evaluation team, owning evaluation coverage for Dialpad's Agentic AI systems alongside our existing evaluation lead
- A key focus will be co-owning LLM-judge metric development and calibration, scenario and benchmark dataset curation, and structured error analysis to support release-readiness decisions for our agentic voice and chat solutions
- This position reports to the manager of the AI Evaluation team and has the opportunity to be based in our Vancouver office
- You will design and execute validation strategies for agentic, NLP, and speech workflows across staging, beta, and release candidates
- You will build, run, and improve regression evaluations, A/B comparisons, and red teaming analyses to determine whether product and model changes are ready to move forward
- You will co-own LLM-judge metric development, calibration, and prompt refinement across evaluation dimensions
- You will create, configure, and monitor data annotation jobs to keep evaluation and calibration datasets fed on schedule
- You will develop and maintain QA tooling, notebooks, and pipeline components that make recurring evaluations scalable and reusable across teams
- You will investigate bugs, triage issues, and decide whether problems should become engineering escalations, test set additions, or follow-up analysis
- You will collaborate with cross-functional teams, including applied science, engineering, and Product QA
Our take
Dialpad offers an AI-driven cloud communications platform for business teams. It was founded to streamline communications across entire businesses with a fast set-up and hassle-free deployment.
As well as hosting in-house video, meet and text communications with automated note-taking and transcription, the platform also features AI-powered customer service, sales and voice intelligence insights. Dialpad has impressed customers such as Asana, Stripe, TED and Uber with its smooth integration capabilities with co-working platforms like Slack, Google Workspace, Salesforce CRM and Microsoft Teams.
The Unicorn company's funding has been heavily invested in the continued R&D of AI technologies, delivering numerous industry-first features. Dialpad has also expanded globally, opening offices in Europe and Asia and extending its global footprint via additional investments internationally. In 2026, it released its Agentic AI platform, which will help businesses to tailor and fine-tune their agentic AI capabilities, indicating that its doubling down on this sector of its business.
Freddie
Company Specialist at Welcome to the Jungle
Insights
8% employee growth in 12 months
Company
Funding (last 2 of 9 rounds)
Dec 2022
$50m
LATE VC
Dec 2021
$170m
LATE VC
Total funding: $450m
Company benefits
- Company stock options
- Continued education stipend
- Cell phone, home internet, and gym membership stipend
- Work from home opportunities
Company HQ
San Ramon, CA
Leadership
Craig Walker
(Founder & CEO)Having studied Law at UCB, Craig worked as an Attorney for Brobeck Phleger & Harrison, before moving to General Counsel at TeleSoft Partners. He spent almost 4 years at Google Ventures before founding Dialpad in January 2011.
Salary benchmarks
We don't have enough data yet to provide salary benchmarks for this role.
Submit your salary to help other candidates with crowdsourced salary estimates.
Share this job
View 31 more jobs at Dialpad