Senior ML Engineer, Cast AI
Kimchi, LLM Inference Optimization
+ Equity Options
More information about location
Kubernetes optimization platform
Posted today
Kubernetes optimization platform
201-500 employees
Posted today
+ Equity Options
More information about location
201-500 employees
Company mission
To pioneer a brighter future of cloud computing.
Role
Who you are
- 5+ years building real ML systems, with a portfolio that shows depth in inference or training infrastructure (not just model training notebooks)
- Strong Python – production services, not scripts
- Hands-on experience with at least one of vLLM, SGLang, or TensorRT-LLM, and a working mental model of why an inference engine performs the way it does on a given GPU
- Fluency with quantization tradeoffs – you’ve measured quality regressions, not just compression ratios
- Comfort with distributed systems: collective communication, sharding strategies, and the practical failure modes of multi-GPU and multi-node setups
- A bias toward measurement. You instrument before you optimize, and you can tell the difference between a real win and a benchmark artifact
- Self-direction. This role comes with a wide mandate; you should be excited by that, not unsettled by it
What the job involves
- Throughput. Latency. KV cache utilization. Move those three numbers in the right direction, and two things happen: customers get faster, cheaper inference, and our margins improve. That’s the entire thesis of this role
- Every kernel you tune, every quantization scheme you ship, every scheduler tweak you land shows up directly in a customer’s p99 and on our P&L
- This is a high-impact seat. It is also a high-autonomy seat as you’ll be given the room to lead the technical direction of inference optimization at Kimchi, not execute someone else’s roadmap
- The problem: running LLMs in production is a moving target. The “right” model and serving configuration for a workload depend on traffic shape, sequence-length distribution, batch dynamics, GPU SKU, memory bandwidth, quantization tolerance, and a dozen other variables that shift week to week
- Most teams pick a model once, over-provision GPUs, and absorb the cost. Kimchi is the system that makes that decision automatically – continuously matching workloads to the most cost-efficient, best-performing LLM and serving configuration on a customer’s infrastructure
- We’re building the optimization layer between the model and the hardware, and we need engineers who understand both sides deeply
- Push throughput. Continuous batching, speculative decoding, chunked prefill, kernel-level tuning across vLLM, SGLang, and TensorRT-LLM. Find the ceiling on each GPU SKU, then raise it
- Cut latency. Attack TTFT and TPOT separately. Profile, identify the actual bottleneck (compute, memory bandwidth, scheduling, networking), and fix it – not the bottleneck you assumed
- Get more out of the KV cache. Paged attention, prefix caching, eviction policies, cache reuse across requests, quantized KV. This is where a lot of the unrealized throughput lives, and it’s an area you’ll own
- Quantize without regressing quality. INT8, INT4, FP8 across weights, activations, and KV. Empirical work: measure quality on real workloads, not just perplexity benchmarks
- Shrink cold starts and memory footprint. Faster init, smarter weight loading, tighter memory accounting – the difference between a model that scales and one that doesn’t
- Scale across nodes. Distributed inference topologies, network-aware placement, checkpointing strategies that don’t bottleneck on storage or interconnect
- Set the technical direction. Decide what we benchmark, what we adopt, and what we build ourselves. Bring the team along with strong writeups and reproducible experiments
- Stack:
- Python; vLLM; SGLang; TensorRT-LLM; PyTorch; CUDA-adjacent tooling; Kubernetes; gRP; ClickHouse; PostgreSQL; GCP Pub/Sub; AWS / GCP / Azure; GitLab CI; ArgoCD; Prometheus; Grafana; Loki; Tempo
Application process
- Screening call with Recruiter
- Hiring Manager interview
- Technical interview (system design)
- Live coding
- Culture Check interview with an executive
Our take
Managing cloud deployments effectively to control costs has become a priority for organizations of all sizes. Cast AI drives cloud optimization and cost reduction by using AI and machine learning to analyze a company’s cloud environment to identify the resources required for maximum performance.
Cast AI’s platform supports all three major cloud service providers; Microsoft Azure, Google Cloud, and Amazon Web Services. When cloud providers have unused machines they are typically discounted by 60-70%, however, if AWS (for example) needs that machine back, they give little warning before reclaiming it. Cast AI is able to automatically detect this, and immediately replace it, meaning companies using its platform will never experience an outage of discounted machines.
As more enterprises embrace cloud and AI technologies, Cast AI has continued to strengthen its position through global expansion and enterprise adoption. The company recently introduced OMNI Compute, a unified compute control plane designed to optimize workloads and PGU usage across cloud providers, while also surpassing a $1B valuation following strategic investment to support its long-term growth and international expansion.
Steph
Company Specialist at Welcome to the Jungle
Insights
51% employee growth in 12 months
Company
Funding (last 2 of 5 rounds)
Apr 2025
$108m
SERIES C
Nov 2023
$35m
SERIES B
Total funding: $180.7m
Company benefits
- Remote work: Enjoy the flexibility of remote-first work. While we have offices, our team is globally distributed, and remote work is fully supported
- Company stock options: Become a shareholder from day one. We offer stock options to all team members as part of our commitment to shared success.
- Annual Hackathon: Innovation is in our DNA. Join our global hackathon to collaborate, build, and explore new ideas beyond your usual role.
- Health insurance: Your well-being matters. We offer comprehensive health coverage and wellness support.
- Flexible working hours: Design a work schedule that fits your lifestyle. We trust you to manage your time effectively.
- Home office allowance: We help you create a comfortable, productive workspace with a dedicated home office stipend.
- Bonus days off: Recharge with extra paid time off beyond standard holidays – because work-life balance matters.
- Learning & development budget: Invest in yourself. We provide a budget for courses, conferences, and other learning opportunities.
- Company events & team building: From local meetups to virtual socials and team retreats, we make time to connect and celebrate.
Company HQ
Downtown Miami, Miami, FL
Leadership
Yuri Frayman
(Co-Founder & CEO)Co-founder & former CEO of ZENEDGE, Frayman Group, Viewdle, and LegalKEY Technologies. Founder & former Executive Chairman of CUJO AI. Current Director of Blumarine Group, and is a Board Member/Advisor at Koop Technologies, Secerus, and Perthera AI.
Leon Kuperman
(Co-Founder & CTO)Co-founder & former CTO of ZENEDGE, Co-founder of HostenPCI, and Founder & previous CTO of Truition. Previous CTO & Technology Advisor at CUJO AI, Vice President of Security Products OCI at Oracle, and was the President & CTO of Bidz.com. Current Board Member for OXYGEN.
Laurent Gil
(Co-Founder & President)Co-founder & former CPO of ZENEDGE. Previous CEO of Viewdle, CFO of WiPlug, and Director of IdeaCapital. Worked as a Group Architect at Oracle, and was an Advisor for both We Heart It and Cel-Fi.
Share this job
View 4 more jobs at Cast AI