Staff / Senior Staff AI Infra Engineer, Together AI
Inference Infrastructure
$250-320k
Plus equity
4 days a week in office
Cloud platform for building generative AI
Be an early applicant
Cloud platform for building generative AI
201-500 employees
Be an early applicant
$250-320k
Plus equity
4 days a week in office
201-500 employees
Company mission
To empower developers and researchers at organizations of all sizes to train, fine-tune, and deploy generative AI models.
Role
Who you are
- Experience operating a provisioning or host-lifecycle pipeline in production, comfortable troubleshooting stuck or failed state transitions (discovery, bring-up, GPU driver/CUDA validation, health checks, decommission/RMA) at fleet scale, and translating real failure patterns back into improvements to the state machine’s design
- Deep experience owning production reliability for large-scale distributed systems; GPU/ML-serving infrastructure experience is a strong plus
- Hands-on experience troubleshooting InfiniBand/RDMA networking issues, ideally in a multi-node training or disaggregated-inference context — comfortable reading IB port counters and subnet manager logs, not just application metrics
- Familiarity with the operational realities of multi-tenant model serving (LoRA/adapter-based serving) and with running differentiated SLOs across model pools of different priority
- A demonstrated track record of eliminating operational toil through automation and tooling you built yourself, not just process or documentation
- Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems
- Comfort with event-driven architectures, backpressure/circuit-breaking design, and treating observability (metrics, logs, traces) as something built into a system, not added after an incident
- A software-engineering mindset first: you write and ship production code, and you think about GPU infrastructure as software to be engineered, not a fleet to be manually operated
What the job involves
- We're looking for a software engineer, reliability-minded but still a builder, to own what happens in production to the Kubernetes-native control plane that provisions and runs our GPU inference fleet
- The platform team builds the manifest-driven API and controllers that let the inference team declare what they need, a cluster, a model deployment, a capacity change, and get it reconciled underneath
- Your job is to keep that promise true in the real world: triaging InfiniBand issues when prefill and decode nodes stall exchanging KV cache over RDMA, diagnosing hero-model and LoRA-serving faults that don't show up in generic health checks, and extending the automated self-healing system so each new failure mode becomes a permanent fix instead of a recurring page
- The core value we're after is keeping the decoupling real even when things break, and keeping the fleet efficient without that efficiency becoming a new source of instability
- You'll own SLOs and error budgets for inference serving, lead incident response, and turn what postmortems teach you into actual changes in the controllers and automation, not just updated runbooks
- You'll also work the reliability side of capacity efficiency, since a mistuned defragmentation or bin-packing pass is itself a production risk
- Strong candidates have hands-on production-reliability experience on large-scale distributed or GPU-serving systems, are comfortable troubleshooting InfiniBand/RDMA and Kubernetes controller/CRD-based platforms, and think about GPU infrastructure, and the tools that keep it reliable, as software to be engineered
- A product mindset applies: the on-call experience, the clarity of an alert, and the judgment built into an automated remediation are things you ship and iterate on
- Operate the provisioning state machine in production: own the health of the host lifecycle pipeline, discovery, inference bring-up, GPU driver/CUDA validation, health checks, decommission/RMA, triaging stuck or failed state transitions at fleet scale and feeding real failure patterns back into how the states and transitions are designed
- Triage and extend automated self-healing: when the node-scanning and remediation platform misses something, a new failure mode, an InfiniBand issue in disaggregated prefill/decode serving, a GPU Xid variant, a hero-model or LoRA-specific fault, diagnose it live, drive the fix to production, and turn it into a new detection/remediation plugin so it's automatic the next time
- Own end-to-end reliability of the pipeline: hold the SLOs and error budgets for provisioning and fleet health, lead incident response when idempotency, retry, rollback, or drift assumptions break down in practice, and drive the systemic fix, not just the hotfix
- Partner with the inference/ML platform team from the production side: be the one who knows how real cluster topologies, interconnects, and scheduling constraints actually fail in the field, and feed those failure modes back as first-class requirements into the platform's abstractions
- Work alongside machine learning researchers, engineers, and product managers to bring our latest technologies into production, enabling advanced research through awesome engineering
- Introduce new techniques, tools, and architecture that improve the performance, latency, throughput, and efficiency of our model inference stack
- Build tools to give us visibility into our bottlenecks and sources of instability, then design and implement solutions to address the highest-priority issues
- Lead reliability cross-functionally: build coalition across infra, ML platform, and product teams around shared reliability priorities, manage stakeholders through incidents and roadmap trade-offs, and represent the reliability point of view in planning so it shapes decisions before they ship, not after an outage
Our take
Founded in 2022, Together AI is a research-driven AI company that operates a cloud platform for building and running generative AI. The company also contributes to open-source research, models and datasets to advance the use of machine learning.
Together AI's platform enables developers, researchers and businesses to train, fine-tune and deploy generative AI for their needs. With Generative AI gaining a lot of attention recently, the company has some high-profile competitors such as ChatGPT, but its focus on research makes it stand out.
The company has had an incredibly strong start, evidenced by its substantial funding and Unicorn status. It is using these funds to expand its AI Acceleration Cloud product and prepare a large deployment of its GPUs. It will do this whilst securing its position as an open-source model-building platform, with an expectation of 50x company growth over the next five years.
Steph
Company Specialist at Welcome to the Jungle
Company
Funding (last 2 of 5 rounds)
Jul 2026
$800m
SERIES C
Feb 2025
$305m
SERIES B
Total funding: $1.3bn
Company benefits
- Competitive health insurance plans
- Dental and vision insurance
- Pre-tax flexible spending accounts
- Mental health support and services
- Income protection & retirement
- 401(k) plan
- AD&D insurance
- Life insurance
- STD & LTD insurance
- Monthly team lunches
- Flexible time off policy
- Team-driven celebrations and events
- Monthly commuting stipend + pre-tax bene
Company HQ
Design District, San Francisco, CA
Leadership
Ce Zhang
(Co-Founder & CTO)Previously Professor at ETH Zurich, Postdoc Researcher at Stanford University and Research Assistant at University of Wisconsin-Madison.
Vipul Ved Prakash
(Co-Founder & CEO)Previously Senior Director at Apple, CEO and CTO at Topsy and Founder at Cloudmark.
Salary benchmarks
We don't have enough data yet to provide salary benchmarks for this role.
Submit your salary to help other candidates with crowdsourced salary estimates.
Share this job
View 64 more jobs at Together AI