AI Engineer, Mistral AI

• 3-4 years of experience; backgrounds that fit well include ML engineers moving closer to product, or software engineers with real AI/ML production experience
• Strong TypeScript or Python skills - we have both tracks depending on team fit
• Production LLM experience: prompts, tool/function calling, system prompts
• Hands-on with evals and A/B testing; you can design metrics, not just run them
• Comfortable implementing directly in product code, not only notebooks
• Observability experience: logging, tracing, dashboards, alerting
• Product mindset: form hypotheses, run experiments, interpret results, ship
• Clear communication, autonomous, and oriented toward production impact over experimentation for its own sake

Desirable

• Embedded directly in the product Team, you will build the evaluation and A/B testing framework, add end-to-end observability, and run a reliable model release process
• Work with Science to ship measurable improvements to quality, latency, safety, and reliability
• You'll build and maintain an LLM evaluation framework (reference tests, heuristics, model-graded checks)
• Define and track metrics: task success, helpfulness, hallucination proxies, safety flags, latency/cost
• Run A/B tests for prompts, models, and system prompts, analyze results, recommend rollout or rollback
• Set up observability for LLM calls: structured logging, tracing, dashboards, alerts
• Operate the model release: canary and shadow traffic, sign-offs, SLO-based rollback criteria, regression detection
• Improve core behaviors: memory write/retrieve policies and evals, intent classification, follow-ups, routing, tool-call reliability
• Create templates and docs so other teams can author evals and ship safely
• Partner with Science, diagnose regressions and lead post-mortems
• We strongly believe in the value of in-person collaboration to foster strong relationships and seamless communication within our team

• Introduction call - 30 min
• Hiring Manager interview - 30 min
• Technical Rounds:
• Live-coding Interview - 45 min
• AI Engineering Interview - 45 min
• Culture-fit discussion - 30 min
• References