Senior Staff Inference Performance Engineer, d-Matrix

• 8+ years of hands-on performance or systems engineering, including resolving hard regressions that crossed multiple layers of a stack
• Track record of finding performance overhead through profiling in unfamiliar, sparsely documented systems, valued more than experience with any specific vendor stack or accelerator
• Deep systems fundamentals, including Linux internals, NUMA, CPU pinning and cpusets, PCIe, memory and cache behavior, and network paths
• Experience running performance-sensitive workloads on Kubernetes, including identifying when scheduler defaults, network overlays, or orchestration abstractions cost latency or throughput, and closing the gap
• Working knowledge of LLM inference serving, including batching, KV-cache, prefill/decode, quantization, and speculative decoding
• Proficiency in Python and C/C++, sufficient to instrument, debug, and patch code across the stack
• Clear technical writing that hardware, firmware, or platform teams can act on without context from your part of the stack
• Sound judgment about ownership, with examples of correctly handing off problems that were not yours to fix
• Fluency with AI and agentic tools for investigation, code search, log triage, and drafting instrumentation, with the discipline to verify output against real data

Desirable

• The Inference Cloud Engineering team builds the deployment and orchestration technology behind d-Matrix’s inference services: an internal cloud where teams test and tune models, and a customer-facing Cloud with self-service inference endpoints and compute
• As the team’s Inference Performance Engineer, you will own end-to-end performance of the workloads we deploy, from the moment a request arrives to the moment a result is returned
• The standard is that a workload on our orchestrated platform should perform at least as well as it does on bare metal
• Today’s workloads include speculative decoding and attention-FFN disaggregation across d-Matrix Corsair accelerators and GPUs, with more heterogeneous architectures coming
• Because there is no fixed system to master, the job is a method: profile first, form a hypothesis, test it, and then change the system
• You will own the outcome, not every fix
• Many root causes sit with other teams, so you will fix what you understand well and hand off the rest with evidence the owning team can act on
• Diagnose performance problems across the full serving path, from orchestration (scheduling, placement, networking, autoscaling) to inference internals (batching, KV-cache, prefill/decode, serving framework, driver and runtime, interconnect, host OS and NUMA, accelerator)
• Isolate each problem to a specific layer and owner, then decide whether to fix it directly, route it to the owning team, or both, with a workaround now and a tracked handoff for the permanent fix
• Write minimal, reproducible problem statements for cross-stack findings so owning teams can act without repeating the investigation
• Protect throughput, latency, and stability targets for orchestrated workloads by finding and closing gaps against bare metal, such as placement overhead, non-performance power defaults, and extra network hops
• Assess what it takes to scale from single deployments to fleet operation, including routing, placement, and shared-resource contention
• Capture each workload’s node requirements (CPU placement, memory and NUMA locality, network path) in the orchestration layer’s configuration so performant deployment is the default
• Build lightweight telemetry and benchmarks that reach into prototype stacks to isolate or hand off problems
• Turn recurring investigations into playbooks and mentor engineers in the profiling-first method