Senior Machine Learning Engineer , Ambience Healthcare

• 5+ years in production ML, research engineering, or applied AI
• Have built a consequential production AI system or materially improved model behavior in production
• Strong understanding of modern LLMs, transformers, and production AI systems
• Experienced designing evaluations for LLMs, agents, or other complex AI systems
• Can turn ambiguous quality problems into measurable dimensions, datasets, and experiments
• Familiar with challenges such as grader bias, leakage, misleading aggregate metrics, regression detection, and offline-online mismatch
• Experience building production systems involving multiple models, tools, retrieval, context, state, routing, or orchestration
• Understands reliability and failure modes in complex AI workflows, not just individual model calls
• Proficient in Python and modern ML frameworks; PyTorch preferred
• Comfortable with deployment, observability, CI/CD, and containerized systems
• Still highly hands-on: writes code, inspects traces, analyzes failures, and debugs production systems
• Skilled at building high-quality datasets and feedback loops
• Experienced using production failures, user feedback, and active learning to improve model and system quality
• Able to work closely with clinicians, product managers, and fellow engineers
• Strong communicator who can simplify complex AI concepts for diverse audiences
• Comfortable owning ambiguous technical problems and driving them to measurable outcomes

Desirable

• As a Senior Machine Learning Engineer at Ambience, you will build and improve the AI systems that power our clinical products
• You’ll own complex projects end-to-end, from diagnosing production failures and designing evaluations to building, deploying, and iterating on model and agentic systems
• This is a highly hands-on role with significant technical ownership
• You’ll work closely with clinicians, product managers, and fellow engineers to translate cutting-edge research into reliable, production-grade AI systems
• Build Trustworthy AI Evaluation Systems: Design and own evaluation pipelines for LLM and agentic systems, combining automated graders, regression testing, production feedback, and human evaluation to measure real product quality
• Improve Production Model Behavior: Diagnose high-impact failure modes and test improvements across prompting, retrieval, context, routing, data, fine-tuning, or other model and system interventions
• Build Agentic AI Systems: Develop production systems involving tool use, retrieval, context and state management, routing, orchestration, tracing, and failure recovery
• Build Data and Improvement Flywheels: Turn production failures and user feedback into better datasets, evaluations, and model behavior through active learning and systematic iteration
• Stay at the Cutting Edge: Distill insights from recent research in LLMs, agents, NLP, speech, and multimodal AI and translate promising ideas into practical experiments
• Own AI Systems End-to-End: Work across models, data, evaluation, orchestration, serving, and observability, while remaining deeply hands-on in code and production debugging