Software Engineer, Apollo Research

• You must have experience writing production-quality python co
• We are looking for strong generalist software engineers with a track record of taking ownership
• Candidates may demonstrate these skills in different ways
• Led the development of a successful software tool or product over an extended period (e.g. 1 year or more)
• Started and built the tech stack for a company
• Worked your way up in a large organisation, repeatedly gaining more responsibility and influencing a large part of the codebase
• Authored and/or maintained a popular open-source tool or library
• 5+ years of professional software engineering experience
• Experience working with AWS

Desirable

• We're seeking a Software Engineer who will enhance our capability to evaluate Large Language Models (LLMs) through building critical tools and libraries for our Evals team
• Your work will directly impact our mission to make AI systems safer and more aligned
• What You'll Accomplish in Your First Year
• Accelerate our frontier LLM evaluations research by leading the design and implementation of software libraries and tools that underpin our end-to-end research workflows
• Ensure the reliability of our experimental results by building tools that identify subtle changes in LLM behavior and maintain integrity across our research
• Shape the vision for our internal software platform, leading key decisions about how researchers will run workloads, interact with data, analyze results, and share insights
• Increase team productivity by providing design guidance, debugging, and technical support to unblock researchers and enable them to focus on their core research
• Build expertise working with state of the art (SOTA) AI systems and tackling the unique challenges posed when building software around them
• Rapidly prototype and iterate on internal tools and libraries for building and running frontier language model evaluations
• Lead the development of major features from ideation to implementation
• Collaboratively define and shape the software roadmap and priorities
• Establish and advocate for good software design practices and codebase health
• Establish design patterns for new types of evaluations
• Build LLM agents that automate our internal software development and research
• Work closely with researchers to understand what challenges they face
• Assist researchers with implementation and debugging of research code
• Communicate clearly about technical decisions and tradeoffs
• Representative Projects:
• Implement an internal job orchestration tool which allows researchers to run evals on remote machines
• Build out an eval runs database which stores all historical results in a queryable format
• Implement LLM agents to automate internal software engineering and research tasks
• Design and implement research tools for loading, viewing and interacting with transcripts from eval runs
• Establish internal patterns and conventions for building new types of evaluations within the Inspect framework
• Optimize the CI pipeline to reduce execution time and eliminate flaky tests
• The current evals team consists of Mikita Balesni, Jérémy Scheurer, Alex Meinke, Rusheb Shah, Bronson Schoen, Andrei Matveiakin, Felix Hofstätter, and Axel Højmark. Marius Hobbhahn manages and advises the team, though team members lead individual projects.
• You would work closely with Rusheb and Andrei, who are the full-time software engineers on the evals team, but you would also interact a lot with everyone else. You can find our full team here

• How to apply: Please complete the application form with your CV. The provision of a cover letter is optional but not necessary. Please also feel free to share links to relevant work samples
• Applications deadline: The date for submissions has been extended to 23 May 2025. However, we review applications on a rolling basis and encourage early submissions.
• About the interview process: Our multi-stage process includes a screening interview, a take-home test (approx. 2 hours), 3 technical interviews, and a final interview with Marius (CEO)
• The technical interviews will be closely related to tasks the candidate would do on the job
• There are no leetcode-style general coding interviews
• If you want to prepare for the interviews, we suggest working on hands-on LLM evals projects (e.g. as suggested in our starter guide), such as building LM agent evaluations in Inspect
• Start Date: Target of 2-3 months after the first interview