Senior Observability & Telemetry Engineer, Submer
Radian Arc
Cooling & automation for data centers
Be an early applicant
Cooling & automation for data centers
101-200 employees
Be an early applicant
101-200 employees
Company mission
To create sustainable, efficient, and planet-friendly data centers and other energy-intense environments.
Role
Who you are
- Proven experience operating large distributed infrastructure platforms
- Strong background in observability systems and telemetry pipelines
- Experience building metrics, logging, tracing, alerting, and dashboards at production scale
- Strong programming skills in Go, Python, or Rust
- Experience with large-scale time-series data platforms
- Experience with large-scale GPU cloud platforms, HPC environments, or AI infrastructure
- Experience monitoring AI workloads such as training or inference clusters
- Deep understanding of distributed systems observability
- Familiarity with cloud-native infrastructure such as Kubernetes, automation, and CI/CD
- Experience operating observability systems for high-performance or large-scale environments
- Experience monitoring complex networking environments
- Familiarity with telemetry protocols such as gNMI, SNMP, and streaming telemetry
- Experience integrating network and system telemetry into centralized monitoring platforms
- Strong data analysis capabilities
- Ability to interpret complex telemetry signals and translate them into actionable insights
- Ability to diagnose performance issues across distributed systems
What the job involves
- Start: ASAP
- Type of Contract: Permanent, full-time
- Mission: Design and build the observability platform that powers visibility, reliability, and performance insights for large-scale GPU cloud infrastructure as well as smaller edge deployments
- This role is responsible for designing and implementing key parts of the observability architecture across the platform, enabling engineering, operations, and customers to understand system behavior in real time across distributed AI workloads, GPU clusters, networking fabrics, storage systems, and edge inference environments
- You will design and operate low-latency, high-scale telemetry pipelines that collect, process, and analyze metrics, logs, and traces from infrastructure running across core datacenter clusters and smaller edge deployments
- The platform you build will support internal operations, automated reliability mechanisms, and customer-facing observability experiences
- As a senior engineer, you will lead delivery of major observability initiatives, contribute to the evolution of telemetry standards and SLO implementation, and work with other teams to ensure observability is effectively integrated into the platform architecture from infrastructure to application layers
- You will collaborate closely with infrastructure, networking, storage, and platform engineering teams to provide clear visibility into performance bottlenecks, infrastructure degradation, and distributed workload behavior across both hyperscale GPU environments and smaller edge installations
- This role contributes directly to improving platform reliability by analyzing production telemetry, identifying systemic issues, and driving improvements in performance, efficiency, and operational stability across the stack
- Design and implement scalable telemetry pipelines for metrics, logs, and traces across distributed GPU infrastructure
- Architect observability systems capable of ingesting high-cardinality telemetry from thousands of nodes and services
- Build and operate telemetry storage systems optimized for large-scale time-series and event data
- Contribute to observability standards across services, including metrics, tracing instrumentation, logging, and SLO implementation
- Build visibility across compute, storage, and networking layers of the platform
- Instrument GPU clusters, inference workloads, and distributed training environments
- Detect infrastructure degradation such as:
- GPU throttling,
- Network congestion,
- Storage latency,
- Hardware degradation
- Implement telemetry pipelines for GPU, CPU, network, and storage performance metrics
- Build dashboards and monitoring tools that expose system health and performance to both internal teams and customers
- Provide insights into workload performance including:
- GPU utilization,
- Storage throughput,
- Network latency,
- Distributed inference performance
- Develop performance analysis tools that help customers understand system bottlenecks
- Develop and maintain network observability platforms
- Build telemetry collectors and exporters using Python or Go
- Ingest telemetry from infrastructure components including:
- NVIDIA Cumulus Linux,
- VyOS routers,
- Citrix NetScaler / WAF
- Design telemetry ingestion pipelines using protocols such as:
- GNMI,
- SNMP,
- Streaming telemetry
- Design advanced alerting and anomaly detection systems
- Contribute to platform SLOs, SLIs, and reliability metrics
- Build automated detection of infrastructure anomalies
- Integrate observability signals with operational workflows and incident management systems
- Participate in on-call rotations supporting platform observability and telemetry infrastructure
- Partner with platform, networking, storage, and compute teams to instrument services
- Work closely with operations teams to improve monitoring and incident response
- Provide guidance and mentorship to engineers on observability best practices
- Promote good observability practices across teams and help engineers adopt effective instrumentation and monitoring patterns
- Technical Stack: Observability and telemetry technologies used across the platform include:
- Observability Framework:
- Prometheus
- OpenTelemetry
- Grafana
- Distributed logging systems
- High-scale telemetry databases, such as ClickHouse or similar
- Hardware and Infrastructure Telemetry
- Redfish / BMC telemetry
- IPMI
- Linux system metrics
- Hardware health monitoring and node lifecycle telemetry:
- NVIDIA GPU Telemetry:
- NVIDIA DCGM
- DCGM Exporter
- NVML
- NVIDIA GPU Operator telemetry stack
- NVSwitch / NVLink telemetry
- AI Workload Telemetry:
- Distributed training telemetry
- Inference latency and throughput metrics
- NCCL communication health
- GPU synchronization latency
- KV-cache access latency for inference workloads
- Dataset loading and storage I/O performance
- Networking Telemetry:
- NVIDIA NetQ
- GNMI streaming telemetry
- SNMP
- Network flow telemetry
- RDMA / RoCE performance monitoring
Our take
Developments in the AI sector have increased the strain on data centres and chips, generating more heat from their processes than ever before. Cooling methods using air and water are not keeping pace with the amount of energy needed to support AI. Submer is on a mission to build data centres and processing plants that are as sustainable as they are efficient.
Submer’s eco-friendly product for data centres and chip processing is built from a coolant of non-flammable, biodegradable and synthetic liquid that physically acts in the same way as water. The racks for servers are stored and operate from within a smart container which further lessens the heat produced by data centres. Submer’s range of products even allows for the coolant to be reused via a heat exchanger.
It’s an inspired solution for an increasingly pressing problem. Data centres are going nowhere, yet the vital need for greener energy solutions puts pressure on these industries. It’s unsurprising then that Submer has raised considerable funding that is being channelled into the expansion of its product. Server component manufacturers are even beginning to build their products to be compatible with Submer’s, indicating the eagerness with which the industry is welcoming the solution.
Freddie
Company Specialist at Welcome to the Jungle
Insights
Company
Funding (last 2 of 6 rounds)
Oct 2024
$55.5m
SERIES C
Jan 2022
$34m
SERIES B
Total funding: $109.6m
Company benefits
- Medical Insurance Plan
- 401k Employee volunteer contribution Plan
- A great work environment characterised by friendliness, international diversity, flexibility, and a hybrid-friendly approach
- You´ll be part of a fast-growing scale-up with a mission to make a positive impact, offering an exciting career evolution
Company HQ
Rubí, Spain
Leadership
Patrick Smets
(CEO)Former COO of the company. Was previously an Associate Partner at McKinsey & Company, CEO at GEM One and worked for Belgian Defence.
Daniel Pope
(CTO & Co-founder)Founder at InferX AI. Previously Director of Architecture and Integration EMEA at Ingram Micro Cloud and Principal Solution Architect at Odin. Was also Senior Solution Architect at Parallels and was a CEO and CTO at Datahouse Internet.
Pol Valls Soler
(CFO & Co-founder)Previously CPO at FXStreet, Workshop Professor and Key Account Manager at La Salle Technova Barcelona and was Co-Founder at Joiner.
Share this job
View 7 more jobs at Submer