Senior AI Workload Platform Engineer, Submer
Radian Arc
Cooling & automation for data centers
Be an early applicant
Cooling & automation for data centers
101-200 employees
Be an early applicant
101-200 employees
Company mission
To create sustainable, efficient, and planet-friendly data centers and other energy-intense environments.
Role
Who you are
- The role combines deep hands-on engineering with ownership of critical orchestration components, including Kubernetes-based compute orchestration, Slurm-based distributed training and batch scheduling, and workflow automation through Argo
- Proven experience working with large-scale distributed compute environments at a neo-cloud, hyperscaler, or HPC provider
- Strong experience with CloudStack internals, including extending and maintaining platform functionality
- Experience operating cloud orchestration platforms in production environments
- Experience running GPU-heavy infrastructure for AI training, inference, or HPC workloads
- Experience maintaining or extending large Java codebases, ideally within infrastructure platforms
- Strong programming skills in Go and Python, with experience building cloud-native platform components
- Experience designing and maintaining control-plane services for infrastructure platforms
- Deep practical knowledge of Kubernetes internals and Slurm scheduling systems
- Experience building or operating compute orchestration layers for large-scale clusters
- Familiarity with workflow orchestration systems such as Argo Workflows
- Familiar with virtual networking and distributed networking technologies such as OVS, OVN, VPC networking, RDMA, RoCE, ECMP, EVPN/VXLAN, and leaf-spine fabrics
- Understanding of GPU virtualization and passthrough mechanisms such as QEMU PCI passthrough and NVIDIA MIG
- Experience working with GPU infrastructure, including passthrough, NVIDIA MIG, scheduling, and lifecycle management of GPUs in distributed clusters
- Able to independently own major compute-orchestration initiatives from design through rollout and operational stabilization
- Comfortable solving difficult implementation and operational problems across CloudStack, Kubernetes, Slurm, and workflow orchestration; improving orchestration quality through code, automation, and practical design decisions; collaborating effectively across compute, networking, storage, and platform teams; and influencing engineering practices through expertise and delivery
- Comfortable mentoring peers and improving implementation quality, documentation, operational workflows, and platform reliability within the compute orchestration domain
What the job involves
- Start: ASAP
- Type of Contract: Permanent, full-time
- Mission: Design, build, and operate the compute orchestration layer powering a GPU-native cloud platform for AI and high-performance workloads. (CloudStack, Kubernetes, Slurm, Argo)
- The platform orchestrates GPU clusters supporting large-scale AI training and inference workloads across distributed compute infrastructure
- This role bridges the current production platform, based on CloudStack, with the next-generation orchestration architecture built around Kubernetes, modern batch scheduling frameworks, and workflow orchestration systems
- You will be responsible for maintaining and evolving the existing CloudStack-based deployments while actively contributing to the design and implementation of the next-generation compute platform supporting distributed AI workloads
- Working closely with networking, storage, and platform engineers, you will help implement the platform primitives that expose GPU infrastructure as a scalable, multi-tenant compute platform
- Maintain the existing CloudStack code base used in current production deployments
- Integrate new upstream CloudStack releases into the internal platform fork
- Perform upgrades of existing customer environments to newer CloudStack versions
- Design and execute safe upgrade paths for running production environments
- Troubleshoot orchestration and provisioning issues in existing deployments
- You'll maintain and troubleshoot CloudStack VPC networking
- Work with and understand CloudStack Debian VPC routers
- Manage networking implementations based on:
- Open vSwitch (OVS)
- OVN
- Improve reliability of network orchestration components
- Manage hypervisor implementations based on:
- KVM
- QEMU
- Maintain and evolve the code responsible for QEMU GPU passthrough, including PCI mapping and exposure of L40S, RTX 6000 Pro, and H200 GPUs to virtual machines
- Design orchestration and scheduling primitives for the next-generation platform based on:
- Kubernetes
- Slurm
- Argo Workflows
- Build orchestration workflows that expose GPU and CPU compute resources to platform users
- Integrate compute orchestration with storage and networking services
- Work closely with networking, storage engineers, and platform software engineers to integrate platform primitives
- Design and implement Kubernetes-based GPU/CPU scheduling infrastructure for multi-tenant AI workloads
- Configure and maintain GPU device plugins and resource allocation mechanisms
- Implement GPU scheduling strategies including:
- GPU partitioning, such as MIG where supported
- Multi-GPU job placement
- Topology-aware scheduling for distributed training and inference
- Design node lifecycle automation for GPU clusters including:
- Node provisioning
- Node draining
- Workload migration
- Implement Kubernetes scheduling extensions where necessary such as custom schedulers or batch schedulers
- Design and operate Slurm-based HPC scheduling environments integrated with Kubernetes clusters
- Implement Slurm compute partitions mapped to Kubernetes-managed GPU/CPU nodes
- Develop mechanisms to submit distributed training, fine tuning, or batch workloads from platform APIs into Slurm clusters
- Implement support for:
- Multi-node distributed GPU training
- Gang scheduling
- GPU topology-aware scheduling
- Build automation for:
- Dynamic Slurm node registration
- Elastic compute capacity
- Node health monitoring and recovery
- Integrate Slurm job lifecycle events with platform orchestration services
- Design and implement workflow orchestration using Argo Workflows
- Develop reusable workflow templates for common platform workloads including:
- AI training pipelines
- Data preprocessing pipelines
- Batch inference workloads
- Platform operational workflows
- Implement DAG-based execution pipelines coordinating compute workloads across Kubernetes and Slurm clusters
- Build workflow primitives that expose platform capabilities to users such as:
- Distributed training workflows
- Model evaluation pipelines
- Batch GPU compute workflows
- Integrate workflow execution with platform APIs and platform user interfaces
- Implement orchestration support for distributed AI workloads including:
- Multi-node training
- Distributed inference
- Large model fine-tuning workloads
- Support execution environments such as:
- PyTorch distributed training
- MPI-based workloads
- Containerized training jobs
- Implement mechanisms to coordinate GPU workloads across nodes with low-latency networking
- Design and maintain mechanisms for multi-tenant GPU resource allocation
- Implement quota and fairness policies for compute workloads
- Develop resource isolation strategies across tenants including:
- Namespace isolation
- Compute quotas
- GPU allocation limits
- Integrate compute orchestration with platform billing and metering systems
- Technical Stack:
- Programming languages
- Java, Python + Bash, SQL for CloudStack-related work
- Go for Kubernetes-related components
- Python
- Orchestration
- CloudStack
- KubeVirt
- Slurm/SUNK
- Kubernetes CRDs and controllers
- Batch scheduling frameworks
- Networking
- OVS
- Linux networking
- VPC networking
- BlueField networking
- Infrastructure
- GPU infrastructure
- Distributed compute clusters
- High-performance networking for distributed AI workloads
Our take
Developments in the AI sector have increased the strain on data centres and chips, generating more heat from their processes than ever before. Cooling methods using air and water are not keeping pace with the amount of energy needed to support AI. Submer is on a mission to build data centres and processing plants that are as sustainable as they are efficient.
Submer’s eco-friendly product for data centres and chip processing is built from a coolant of non-flammable, biodegradable and synthetic liquid that physically acts in the same way as water. The racks for servers are stored and operate from within a smart container which further lessens the heat produced by data centres. Submer’s range of products even allows for the coolant to be reused via a heat exchanger.
It’s an inspired solution for an increasingly pressing problem. Data centres are going nowhere, yet the vital need for greener energy solutions puts pressure on these industries. It’s unsurprising then that Submer has raised considerable funding that is being channelled into the expansion of its product. Server component manufacturers are even beginning to build their products to be compatible with Submer’s, indicating the eagerness with which the industry is welcoming the solution.
Freddie
Company Specialist at Welcome to the Jungle
Insights
Company
Funding (last 2 of 6 rounds)
Oct 2024
$55.5m
SERIES C
Jan 2022
$34m
SERIES B
Total funding: $109.6m
Company benefits
- Medical Insurance Plan
- 401k Employee volunteer contribution Plan
- A great work environment characterised by friendliness, international diversity, flexibility, and a hybrid-friendly approach
- You´ll be part of a fast-growing scale-up with a mission to make a positive impact, offering an exciting career evolution
Company HQ
RubĂ, Spain
Leadership
Patrick Smets
(CEO)Former COO of the company. Was previously an Associate Partner at McKinsey & Company, CEO at GEM One and worked for Belgian Defence.
Daniel Pope
(CTO & Co-founder)Founder at InferX AI. Previously Director of Architecture and Integration EMEA at Ingram Micro Cloud and Principal Solution Architect at Odin. Was also Senior Solution Architect at Parallels and was a CEO and CTO at Datahouse Internet.
Pol Valls Soler
(CFO & Co-founder)Previously CPO at FXStreet, Workshop Professor and Key Account Manager at La Salle Technova Barcelona and was Co-Founder at Joiner.
Share this job
View 7 more jobs at Submer