Senior Machine Learning Engineer, Scale AI

• This role will require an active security clearance or the ability to obtain a security clearance
• Experience in computer vision, deep learning, reinforcement learning, or NLP in production settings
• Strong programming skills in Python; experience with TensorFlow or PyTorch
• Background in algorithms, data structures, and object-oriented programming
• Experience with LLM pipelines, simulation environments, or automated evaluation systems
• Ability to convert research insights into measurable evaluation criteria

Desirable

• The Public Sector ML team at Scale deploys advanced AI systems—including LLMs, agentic models, and multimodal pipelines—into mission-critical government environments. We build evaluation frameworks that ensure these models operate reliably, safely, and effectively under real-world constraints
• As an ML Engineer, you will design, implement, and scale automated evaluation pipelines that help customers trust and operationalize advanced AI systems across defense, intelligence, and federal missions
• Develop and maintain automated evaluation pipelines for ML models across functional, performance, robustness, and safety metrics, including LLM-judge–based evaluations
• Design test datasets and benchmarks to measure generalization, bias, explainability, and failure modes
• Build evaluation frameworks for LLM agents, including infrastructure for scenario-based and environment-based testing
• Conduct comparative analyses of model architectures, training procedures, and evaluation outcomes
• Implement tools for continuous monitoring, regression testing, and quality assurance for ML systems
• Design and execute stress tests and red-teaming workflows to uncover vulnerabilities and edge cases
• Collaborate with operations teams and subject matter experts to produce high-quality evaluation datasets