Member of Technical Staff, xAI

• 5+ years shipping production software and/or operating distributed infrastructure at scale
• Expert-level knowledge of Linux systems, TCP/IP networking, and systems programming
• Strong coding skills with proven production experience in Rust (strongly preferred) and at least one of Python, Go, or C++
• Deep experience with large-scale distributed systems in on-prem and cloud environments (GCP experience a plus)
• Hands-on expertise with container orchestration (Kubernetes, Borg-class systems, or custom schedulers), container runtimes, and infrastructure-as-code (Puppet/Chef/Ansible/Terraform)
• Intimate understanding of common failure modes in distributed systems and how to mitigate them (blast radius control, failure domains, canaries, chaos engineering, etc.)
• Track record of participating in (or building) effective on-call rotations in high-stakes environments
• Bachelor’s degree in Computer Science, Electrical Engineering, or equivalent real-world experience

Desirable

• We are training some of the largest models in the world on the latest hardware across multiple environments
• To do this reliably at xAI’s pace, we need engineers who have battle-tested experience keeping massive distributed infrastructure up and running 24/7, including on-prem and cloud-based infrastructure
• This is a joint xAI/X role: you will own 24×7 reliability for the world’s largest GPU training superclusters and one of the highest-QPS production systems on the planet (X)
• You will own the availability, performance, and evolution of xAI’s core compute, storage, and networking infrastructure
• This is not an ops-only role — strong coding is a hard requirement
• You will design, implement, and ship systems software, automation, and tooling in Python and/or Rust that directly impact training throughput and cluster utilization
• You will be expected to participate in a team on-call rotation and to contribute to ushering xAI into the next generation of infrastructure management across multiple data centers and cloud environments
• Define and execute the technical strategy for infrastructure reliability and scalability
• Build and maintain the automation, observability, and control planes that keep multi-datacenter, hybrid cloud/on-prem environments healthy
• Lead incident response, deep-dive root cause analysis, and post-mortems that drive real fixes
• Identify, instrument, and eliminate systemic failure patterns (capacity, network, hardware, storage, software)
• Design and implement high-leverage systems software (daemons, controllers, schedulers, etc.) in Python and Rust
• Push the state of the art in large-scale GPU cluster operations and AI workload reliability

• After submitting your application, the team reviews your CV and statement of exceptional work
• If your application passes this stage, you will be invited to an initial interview (45 minutes - 1 hour) during which a member of our technical team will ask some basic questions
• If you clear the initial phone interview, you will enter the main process, which consists of four interviews:
• Coding assessment in a language of your choice
• Deep-dive on distributed systems, networking, and failure modes
• Manager Interview
• 30–45 minute presentation on a large-scale reliability or infrastructure project you personally owned from incident/crises to resolution
• Our goal is to finish the main process within one week
• We don’t rely on recruiters for assessments. Every application is reviewed by a member of our technical team. All interviews will be conducted via Google Meet