Senior LLM Systems Engineer, Risk Labs

• 3+ years of professional software engineering experience in Python, TypeScript, or similar production languages
• Hands-on experience building production systems that use LLMs, agents, retrieval, structured outputs, or model-powered workflows
• Experience designing evaluations, test datasets, regression checks, quality metrics, or manual review loops for AI systems
• Strong debugging ability across APIs, databases, queues, logs, model outputs, and external data sources
• Practical understanding of prompt engineering, tool calling, structured output validation, retrieval, and common LLM failure modes
• Ability to reason carefully about correctness in uncertain or adversarial environments
• High agency, strong ownership, and clear written communication
• If you're excited about this role, even if your experience doesn't align perfectly, we encourage you to apply

Desirable

• We are hiring a Senior LLM Systems Engineer to own and improve the LLM-driven components of our oracle automation stack
• This person will focus on the accuracy, performance, resilience, and operational quality of the systems that use models to reason about wide ranging prediction market rules, evidence, and oracle outcomes
• This is a production systems role, not a research-only or prompt-only role. You will build the evaluations, observability, tooling, fallbacks, and feedback loops that make LLM behavior measurable and dependable in real-world conditions
• What You'll Own:
• LLM Accuracy: improve prompts, model selection, tool usage, structured outputs, retrieval, and evaluation coverage so the system gets more decisions right over time
• System Performance: reduce latency, token usage, and cost while preserving decision quality and operational reliability
• Resilience: design validation, retries, fallbacks, uncertainty handling, and human review paths for ambiguous, adversarial, incomplete, or conflicting inputs
• Evaluation and Monitoring: build datasets, regression tests, dashboards, traces, and review loops that make model quality visible and prevent repeated failures
• Agent and Tooling Architecture: Improve agent orchestration and tool use across internal services, APIs, search workflows, databases, and external data sources
• Production Operations: help debug live issues, investigate regressions, improve runbooks, and reduce repeated operator friction
• What Success Looks Like:
• The oracle automation system handles a wider range of market and resolution scenarios with higher measured accuracy
• LLM quality is tracked through evaluations and regressions instead of judged only through manual spot checks
• Engineers and operators can inspect model behavior, tool usage, reasoning paths, and uncertainty when investigating outcomes
• Latency and cost improve without hiding quality regressions
• The system fails more gracefully when data is missing, tools fail, sources disagree, or cases are genuinely ambiguous
• Tech Stack:
• Our tech stack includes Python, TypeScript, Postgres, GCP, Cloud Run, GitHub Actions, Terraform, React, Node.js, Solidity, and LLM APIs from major model providers