Principal Machine Learning Engineer, Product.ai
Model Evaluation & Verification
$500-600k
OTE
$350-420k
BASE
More information about location
5 days a week in office
Verification layer for AI commerce
Be an early applicant
Verification layer for AI commerce
21-100 employees
Be an early applicant
$500-600k
OTE
$350-420k
BASE
More information about location
5 days a week in office
21-100 employees
Company mission
To separate genuine product knowledge from the flood of AI-generated marketing noise now dominating the internet.
Role
Who you are
- You interrogate every green checkmark. A passing eval is a claim, and claims get challenged, so you ask what the test could not have caught before you ask what it confirmed. You form working models of complex systems on your own. When the model is wrong, you notice and update fast. You write clearly, because clear writing is evidence of clear thought
- You treat agents as leverage you verify rather than an oracle you trust. You can do this job by hand and prove it: hand-grade a judge, hand-build a corpus, hand-verify a claim set. That mastery is exactly what lets you trust, or reject, the verdict an agent hands back. Compute is cheap here. The expensive thing is a redo cycle
- You have probably built an eval harness another team came to depend on, a regression corpus that caught a real failure before users did, an LLM-judge pipeline where you measured the judge's bias instead of trusting it, or a golden set for a system whose output is never the same twice
- Adjacent roads count: scoring systems from fraud, search relevance, or content integrity, and any other domain where ground truth was scarce and the adversary was real. We care about the artifact and the reasoning behind it far more than where you did it
What the job involves
- Everything we sell rests on one question. Is the claim true? Machine verdicts answer it, and they decide what we publish about a product, a price, a code. The constraint on growth is whether the machine verdicts stay right as the volume climbs
- You own that, end to end: the eval science behind every automated verdict, from the judges and thresholds to the exams that grade them. We call that whole verdict chain the Truth Engine, and it decides what we publish as true for millions of shopping claims. You work directly with the founder
- You own the arbitration rulebooks, the rules that settle a contested verdict. You also own the structural check that keeps the truth this company ships from resting on any one person's judgment, including yours. That check is a held-out, human-graded exam the machine is measured against, plus a written authority split with the founder
- You are not joining a team that does this; you are the person who does it. It is also the most durable engineering seat in the building. Every model generation absorbs another layer of surface skill, but grading a verdict no human can check faster than the machine is an open frontier that grows in value for years
- Adversarial robustness, end to end. Merchants, bots, and models all have incentives to game a verdict. A merchant wants its expired code to pass as live, and a bot wants to look like a shopper
- An LLM judge grades its own model family measurably too generously, a self-preference bias you measure and correct rather than assume away. Your verifier has to be structurally harder to fool than any of them
- Calibration with decay. A verdict is more than a yes or a no. It comes with a confidence and a shelf life. A price claim goes stale in days; a materials fact stays true for years. The pipeline has to score how sure the machine is, per claim class, and admit when it is not sure
- That means calibrated confidence with principled abstention, because a confidently wrong verdict is worse than no verdict at all
- Golden sets: grading the grader. The hardest problem in the seat is meta-evaluation, measuring a judge whose judgments no human can make faster than the machine
- The answer is a held-out, human-graded exam the model cannot see, cannot train on, and cannot game. Label quality, inter-rater agreement, contamination control, refresh cadence. Designing that exam is the science
- The verification ladder. Every claim climbs one, from unverified to machine-settled to human-graded, and each rung is cheaper than the one above and harder to fool than the one below. What survives feeds Axiomatic Intelligence, our stress-tested knowledge layer, where a proven verdict gets reused instead of re-litigated
- Evals as the ship gate. Eval-driven development is how engineering works here. Agent loops write the code and the content; your corpora and gates decide whether a loop's output ships or gets rejected. The eval is the spec
- Cortex, the shared AI brain. The governed substrate the company runs on and our products are built on; it answers its own questions from thousands of internal documents. Your verification keeps those answers trustworthy
- Scale, under model churn. Hundreds of thousands of merchants, millions of claims. An eval method that works on a hundred cases and falls over at a million is a demo. And the ground moves under it. Each model generation resets what judges can do and what your exams still discriminate, so the harness gets rebuilt while it runs
- The judge pipeline. The machine verdicts that decide what we claim is true about shopping: the models, the thresholds, and the rulebooks behind every automated verdict. Your number is the share of verdicts machines settle. You take it from a minority to the large majority and keep a high accuracy floor against the human-graded exam
- Calibration and abstention. How sure the machine is, per claim class, each with its own rate of decay. A judge that reports 95% confidence and is right 70% of the time is miscalibrated, and that bug is yours
- The golden sets. You grow them and keep them clean, which means label quality, contamination control, and refreshing before they saturate. You also give them teeth. When a verdict is disputed, the exam decides, not the loudest engineer in the room
- The accuracy floor other people now pay for. Developers and AI agents buy our verified commerce data on metered keys. Anonymous access closed in August, and the free legacy codes are closing behind it. A miscalibrated judge is no longer an internal quality bug. It is a defect a paying customer sees. You own the floor that makes the data sellable
- Your seat charter. Within your first quarter you co-sign a charter for this seat. It names one machine-checkable number that proves the seat is working, and a written split of what you decide freely versus what you bring to the founder to decide
- You will use the craft you already own (judge design, golden sets, calibration, regression corpora, statistical rigor) and grow into the layer above: eval-gated agent orchestration, adversarial verification at commerce scale, and adjudication design for judgments no single grader can settle
Application process
- We don't run traditional AI engineering interviews. Every stage is demonstrated performance on work-relevant tasks
- Async video screen. About 15 minutes, on your own time — it replaces the recruiter screen. We want to see how you think, not how you present
- Calls with company stakeholders. Short conversations with the team
- Conversation with the founder. How you model the system above, where you push back, and whether you can hold the argument live
- Paid work trial. One week of paid, real verification work in our environment — live loops, live corpora, real verdicts. It is paid because your time is worth paying for. We watch how you get grounded, whether you write the spec before the build, how you verify what your agents produce, and whether your self-assessment is honest
Our take
Artificial intelligence has come on strides in recent years, but many assistants still struggle with relatively simple tasks. For example, AI-powered shopping assistants have frequently been found to supply conflicting information regarding prices, product specifications and more.
Product.ai is building what it calls the verification layer for AI commerce, to combat inaccurate and misleading product overviews. The platform can do things like check whether coupons actually work, verify the latest prices, and decipher whether claims about products are true.
Formerly known as Demand.io, the company's earlier focus was on creating innovative eCommerce experiences, but its rebrand to Product.ai in 2026 cemented its pivot to creating the truth layer for eCommerce. This mission answers a growing problem for consumers, as increasing AI marketing noise drowns out true reviews - meaning there should be ample opportunity for Product.ai to make an impact.
Steph
Company Specialist at Welcome to the Jungle
Company
Company benefits
- Full health coverage: All premiums covered for you and your family. Medical, dental, vision.
- Daily catered lunch: For the LA team. One less thing to think about during the workday.
- Unlimited PTO: That we actually expect you to use. Recharge is not optional, it is structural.
- Remote flexibility: An LA headquarters with full remote support. Work where you do your best thinking.
- Free garage parking and commuter assistance
Company HQ
Sawtelle, Los Angeles, CA
Leadership
Michael Quoc
(Founder & CEO)Experience as Senior Product Manager for Yahoo! Inc. from 2002 to 2006 and as Director of New Products from 2006 to 2008.
Brian Stanback
(Chief Architect)A software Engineer with a passion for building highly interactive web applications. Brian has over 15 years of experience as a Web Developer, Software Architect, and Database Engineer.
Jimmy Doheny
(Director of Operations)Jimmy joined Product.ai in 2017 as a Partner Marketing Manager. They then moved into the role of Senior Manager of Revenue Operations in 2019 and Director of Operations in 2021.
Salary benchmarks
We don't have enough data yet to provide salary benchmarks for this role.
Submit your salary to help other candidates with crowdsourced salary estimates.
Share this job
View 2 more jobs at Product.ai