Staff Software Engineer, Harvey

• 10+ years of software engineering experience building large-scale distributed systems
• Experience designing and operating highly available production services
• Strong programming skills in Go, Java, Python, Rust, or C++
• Deep understanding of distributed systems, cloud infrastructure, networking, and observability
• Experience leading technical projects across multiple engineering teams
• Ability to balance long-term architecture with pragmatic execution
• Strong communication and collaboration skills
• Passion for building foundational platforms that enable other engineering teams

Desirable

• As a Staff Software Engineer on the Model Infrastructure team, you'll lead the design and development of the systems that power every AI request at Harvey
• You'll partner closely with AI Research, Product Engineering, Infrastructure, and external model providers to build a platform that is highly reliable, scalable, observable, and efficient
• Lead the design and implementation of Harvey's Model Infrastructure platform
• Build systems to ensure high availability, low latency, and operational excellence for AI inference
• Design and improve Harvey's Unified Model Controller (UMC) and Model Selector platform to automatically detect model degradations and intelligently route traffic based on reliability, latency, quality, compliance, and cost
• Develop systems for model provisioning, capacity management, failover, and traffic engineering across multiple AI providers
• Integrate new model providers and maintain provider APIs and SDKs, enabling Harvey to rapidly adopt emerging frontier models
• Improve observability through health dashboards, alerting, token usage analytics, cost reporting, and end-to-end telemetry
• Partner with Product Engineering to support model launches, experimentation, and proactive monitoring of production AI workloads
• Drive infrastructure efficiency through capacity planning, utilization optimization, and cost visibility
• Collaborate with AI Research to build the infrastructure foundation for future model evaluation, training, and deployment
• Lead cross-functional technical initiatives and mentor engineers across the organization
• What You’ll Build:
• Model Reliability & Operations:
• Model health monitoring
• Automated failover and recovery
• Capacity provisioning
• Operational tooling and incident automation
• Unified Model Controller (UMC):
• Policy-based model routing
• Intelligent Model Selector
• Traffic management
• Reliability and latency optimization
• Provider Platform:
• Multi-provider architecture
• API and SDK integrations
• OpenAI, Anthropic, Azure OpenAI, Fireworks, Baseten, and future providers
• Rapid adoption of new frontier models
• Observability & Cost Platform:
• Token usage analytics
• Cost attribution
• Latency and reliability dashboards
• Capacity forecasting
• Utilization optimization
• AI Platform Foundation:
• Infrastructure supporting model evaluation
• Model deployment and operations
• Future model training platform
• Agent infrastructure and CcaaS