Machine Learning Engineer / Senior Machine Learning Engineer, C3.ai

• BS/MS in Computer Science/Electrical Engineering or equivalent deep, practical experience
• 5+ years of work experience (8+ years for Senior Machine Learning Engineer)
• Proven track record building custom ML/HPC platforms for specialized research (e.g., novel model architectures, time-series, multimodal AI) where commercial cloud tools were insufficient
• Deep expertise with Kubernetes and/or SLURM on GPU clusters, including proficiency with containers, images, and multi-node scheduling
• Strong software development skills in Python and one of Go, C++, or Rust. Comfortable developing controllers/operators, high-performance services, and CLI tooling on Linux
• Practical, hands-on knowledge of distributed ML frameworks (PyTorch DDP/FSDP/ZeRO, DeepSpeed, or JAX/TPU) and performance profiling (NCCL, CUDA basics, I/O performance)
• Experience with object stores, Parquet format, dataset version control, streaming/sharding techniques, and efficient artifact management for checkpoints and logs
• Practical experience with observability (Prometheus/Grafana/OpenTelemetry) and infra-as-code (Terraform/Helm/Ansible)

Desirable

• C3 AI Data Science team is dedicated to pushing the boundaries of what is possible with large-scale AI
• We are seeking a hands-on Machine Learning Engineer to design, build, and operate a bespoke, next-generation research platform dedicated to training novel, large-scale foundation models far beyond conventional LLM recipes
• This is a critical systems role
• You will create the orchestration, secure data pathways, and frictionless developer experience that empowers our researchers to move fast, experiment securely, and scale complex training jobs on heterogeneous GPU clusters
• We are looking for an expert who can solve infrastructure problems where off-the-shelf cloud tools are insufficient
• Design and manage the core research compute cluster, including node layouts, queues/partitions, preemption/fair-share policies, and multi-tenant isolation
• Implement secure access controls for all users and services across the cluster using Kubernetes and/or SLURM
• Build robust branch-to-experiment CI/CD workflows, encompassing templated job creation, config promotion, and integrated version control
• Implement an experiment and metrics tracking system (runs, configs, checkpoints, logs) with searchable lineage to enable frictionless cross-team collaboration and sharing
• Design and integrate auto-checkpointing, artifact retention, and necessary rollout/rollback mechanisms
• Stand up robust dataset registries, ensuring data lineage, versioning, and secure access
• Implement sharding, streaming, and prefetch mechanisms to support efficient TB-scale data corpora access and long-term archival with reproducible rehydration
• Profile NCCL/I/O hotspots, optimize training throughput (mixed precision/AMP, ZeRO/FSDP, kernel fusion, caching)
• Harden training pipelines for scale and resilience, including checkpoint recovery, and tolerance for spot/preemptible instances
• Build opinionated templates, job specifications, and guardrails to ensure researchers can focus on modifying custom training code and recipes without fighting infrastructure bottlenecks