Software Engineer, Baseten

• This role is ideal for someone who thrives in a fast-paced startup environment and is eager to make significant contributions to the exciting field of LLM inference
• Bachelor's, Master's, or Ph.D. degree in Computer Science, Engineering, Mathematics, or related field
• Experience with one or more general-purpose programming languages, such as Python or C++
• Familiarity with LLM optimization techniques (e.g., quantization, speculative decoding, continuous batching)
• Strong familiarity with ML libraries, especially PyTorch, TensorRT, or TensorRT-LLM
• Demonstrated interest and experience in LLMs
• Deep understanding of GPU architecture

Desirable

• We're looking for inference performance engineers who want to make the world's most demanding AI workloads run faster and more efficiently
• You'll work across the stack, from the inference engine and runtime through scheduling, serving, and routing
• Along the way you'll apply techniques like prefill/decode disaggregation, speculative decoding, and KV-cache management
• You'll reason from first principles about where time and memory go, find what's holding performance back, and close the gap
• Your work directly impacts how fast our customers' models run and how efficiently we serve them
• You'll get to work on these types of projects as an Inference Performance engineer:
• Agentic Kernels in Production
• How we built the new fastest API for GLM-5.2
• Live draft model training for speculative decoding
• Making Kimi K3 Tokenization 18x faster
• The Baseten Inference Stack
• Driving model performance optimization
• Implement and productionize cutting-edge inference techniques, working deep in runtime internals. That includes quantization, speculative decoding, KV-cache reuse, chunked prefill, LoRA, guided generation for structured outputs, and custom scheduling and routing algorithms
• Profile and optimize inference end to end, from kernel launch overhead and memory layout up to request scheduling, prefill/decode disaggregation, and cache-aware routing. Run cross-layer investigations, such as tracing a tail-latency regression from request timing through routing and batching down to a kernel
• Turn performance into cost savings. Improve tokens per GPU-hour, raise utilization, and give customers and internal teams clear latency/throughput/cost tradeoffs
• Bring up and tune new model architectures on new hardware quickly, often in the same week they're released
• Build benchmarking frameworks that measure real-world performance across model architectures, batch sizes, sequence lengths, and hardware configurations
• Contribute upstream to open-source inference engines (vLLM, SGLang, TensorRT-LLM), and partner closely with model, infrastructure, and customer-facing teams to ship wins