Senior AI Infrastructure Engineer, Scale AI

• The ideal candidate is a systems expert who thrives on solving the orchestration, networking, and reliability challenges that emerge at massive scale. You will partner closely with researchers to build a seamless, resilient environment that transforms raw compute into breakthrough AI
• 5+ years of experience in backend or infrastructure engineering, with at least 2 years focused on orchestrating ML workloads at scale (100+ GPU nodes)
• Strong programming skills in one or more languages (e.g. Python, Go, Rust, C++)
• Experience with complex compute management systems that cover queueing, quotas, preemption, and gang scheduling
• Experience with distributed training infrastructure, such as EFA, Infiniband, and topology-aware scheduling
• Experience with distributed storage systems (e.g. Lustre, S3) as they relate to training throughput
• Expert-level knowledge of Kubernetes internals (Custom Resources, Operators, Admission Controllers) and how they interact with device plugins for specialized hardware
• Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (e.g., Terraform)
• Proven ability to solve complex problems and work independently in fast-moving environments

Desirable

• As a Software Engineer on the Machine Learning Infrastructure team, you will build the "Operating System" for our large-scale GPU clusters
• You will architect a high-performance training platform that handles the immense complexity of multi-thousand GPU workloads, ensuring every cycle is used efficiently
• Your work directly determines the velocity at which our researchers can train and iterate on the world’s most advanced models
• Architect and scale a multi-tenant orchestration layer that abstracts away the complexity of GPU clusters, ensuring high utilization and seamless job recovery
• Design and implement scheduling primitives to optimize the lifecycle of training jobs
• Develop deep observability and automated health-checking into the training stack to proactively identify and isolate hardware failures
• Evaluate and integrate emerging technologies in the CNCF and AI ecosystem (e.g. Ray, Kueue), making data-driven build vs. buy decisions that balance velocity with long-term maintainability
• Work closely with Finance and Procurement teams to drive our capacity planning process
• Participate in our team’s on call process to ensure the availability of our services
• Own projects end-to-end, from requirements, scoping, design, to implementation, in a highly collaborative and cross-functional environment