Senior/Staff Software Engineer, Pryon

• We need a Senior/Staff Software Engineer with deep high performance computing expertise - someone who has built parallel and distributed systems at scale, not just used them
• Extensive experience in software development, with a proven track record of delivering complex, large-scale systems (8+ years for Senior, 12+ years for Staff)
• Proven experience building distributed systems at 100M+ scale (documents, vectors, or equivalent)
• Deep knowledge of parallel and distributed computing concepts including consensus algorithms, distributed coordination, and fault tolerance
• Hands-on experience with vector databases (pgvector, Pinecone, Weaviate, Milvus, or equivalent)
• Proficiency in systems programming languages such as C++, Go, or Rust
• Experience with parallel programming models (e.g., MPI, OpenMP, CUDA)
• Production experience optimizing GPU workloads for inference including batch optimization, quantization (INT8, FP16), and GPU memory management
• Experience managing large-scale GPU infrastructure (10+ GPUs in production) including cluster orchestration, resource scheduling, and cost optimization
• Deep understanding of GPU architectures (NVIDIA A100/H100, tensor cores) and inference frameworks (vLLM, TensorRT, Triton)
• Deep understanding of memory hierarchies, cache optimization, and NUMA architectures
• Experience with container orchestration (Kubernetes) and distributed computing frameworks (Ray, Dask, Spark, or equivalent)
• Familiarity with performance analysis and optimization tools and techniques (profilers, tracers, benchmarking)
• Strong systems programming background with evidence of performance-critical contributions (open source, papers, or production systems)

Desirable

• As a founding technical contributor to the SCM team, you will design and implement the parallel computing infrastructure that powers our ingestion, retrieval, and inference layers across multi-cloud and on-prem environments
• Design GPU resource allocation and scheduling for embedding generation and inference workloads across multi-tenant deployments; optimize GPU utilization (targeting 80%+ GPU memory utilization) through techniques like batching, quantization, and vLLM deployment
• Build and operate GPU clusters (multi-node, multi-GPU) for production inference workloads; implement GPU monitoring, health checks, and auto-recovery mechanisms; optimize cost-per-inference through efficient GPU scheduling
• Profile and optimize GPU-based inference pipelines for latency (targeting <100ms p95) and throughput (queries/sec per GPU); implement inference optimizations including KV caching, continuous batching, and model parallelism strategies
• Establish performance benchmarking framework, identify bottlenecks through profiling, and implement optimizations achieving 10x+ improvements
• Design and implement distributed ingestion pipeline capable of processing 6M+ documents/hour using technologies like message brokers, MinIO, workflow engines and parallel processing frameworks
• Architect high-performance vector similarity search infrastructure supporting billion-scale embeddings with sub-second query latency
• Architect compute-optimized deployments across AWS (EC2, EKS), GCP (GKE, TPUs), Azure (AKS), and on-premises Kubernetes
• Design, develop, and optimize high-performance, distributed systems and software for scalability and reliability
• Collaborate with research scientists, ML engineers, and platform teams to deliver high-quality, large-scale software
• Drive architectural decisions through RFCs, mentor ML and platform engineers on HPC best practices, establish coding standards for high-performance systems
• Lead the technical design and implementation of major features and components with lasting architectural impact