Open Source AI Engineer, Arize AI
Evals
$150-185k
Plus equity
Machine learning observability platform
Be an early applicant
Machine learning observability platform
21-100 employees
Be an early applicant
$150-185k
Plus equity
21-100 employees
Company mission
To make the world’s AI work and work for the people.
Role
Who you are
- We’re looking for an engineer who’s deeply passionate about AI, loves working in the open, and thrives in a fast-paced environment where “everyone wears multiple hats”
- Open Source Champion: You believe collaboration and community-driven development unlocks the best innovations
- Creative Problem Solver: You enjoy tackling ambiguous challenges and finding elegant technical solutions
- Data & Metrics Driven: You value empirical results, enjoy creating or refining evaluation metrics, and iterate based on real-world feedback
- Technically Curious: You’re always learning—exploring new LLM architectures, prompt engineering strategies, or emerging library standards
- Builder Mindset: You relish the process of taking ideas from initial prototypes to production-ready solutions that delight users
- Hands-on LLM Experience: Familiarity with popular LLM frameworks, prompt engineering techniques, and model fine-tuning
- Strong Programming Skills: Fluent in Python for AI workflows; bonus if you can navigate TypeScript as well
- Evaluation Knowledge: Understanding of core NLP evaluation methods and experience applying or extending them for LLM systems
- Open Source Track Record: Contributions to open source projects, personal GitHub repos with interesting AI demos, or a history of active engagement in developer communities
- ML Observability & Tools: Familiarity with debugging AI applications, exploring embeddings, or building data-heavy dashboards is a plus
What the job involves
- We’re looking for an Open Source AI Engineer to join our growing OSS team to drive the development of new frameworks, metrics, and tooling that help people build, test, and improve LLM tasks
- You’ll play a lead role in shaping how developers measure and understand performance in advanced AI systems, all in the open
- Build LLM Eval Frameworks: Design, architect, and open-source new libraries, pipelines, and APIs that make it simpler to evaluate LLM output quality, consistency, and reliability at scale
- Define Metrics and Benchmarks: Curate golden datasets and develop robust benchmarked metrics that guide data scientists and AI practitioners in optimizing their AI tasks
- Collaborate with the Community: Partner closely with the broader AI open source ecosystem, gather feedback, review pull requests, and steer the direction of the project to address real developer needs
- Prototype and Iterate Rapidly: Experiment with state-of-the-art LLM techniques, turning research into practical developer tooling
- Improve Observability and Debugging: Integrate with our existing platform to surface deeper insights on LLM behavior—help teams quickly diagnose and fix issues such as hallucinations or bias
- Educate and Evangelize: Write blog posts, white papers, tutorials, and documentation to help developers succeed with our open source tools and grow the LLM eval community
Our take
Machine learning is rapidly becoming a fundamental and mainstream automation tool used across many industries. The issue, however, is that monitoring, assessing, and improving ML models is costly, and especially tricky without monitoring capabilities. It’s created a market demand for ML monitoring tools, and Arize’s platform offers a rich suite of them.
The Arize platform gives developers and operators comprehensive oversight of their models with tools including performance tracing, drift detection, and integration and deployment. This provides a crucial guardrail: as more decision-making is handed over to ML in areas like insurance and loans, poor ML model performance can have a catastrophic effect on both individuals and customers alike.
Remarkably, given this development, ML observability platforms aren’t yet in huge supply. Arize startup rivals like Censius and WhyLabs are making strong headway, but a clear winner has by no means yet emerged. Arize has, however, netted some major customers: Spotify, Instacart, Uber, and P&G are a handful of the major clients that will help it lay a strong foundation in what is anticipated to be a highly lucrative market.
Kirsty
Company Specialist at Welcome to the Jungle
Insights
23% employee growth in 12 months
Company
Funding (last 2 of 3 rounds)
Sep 2022
$38m
SERIES B
Sep 2021
$19m
SERIES A
Total funding: $61m
Company benefits
- Remote work opportunities
Company values
- Integrity - Do what you say, say what you do
- Balance - Hold space for what matters to you
- Candor - Be open, be honest, be constructive
- Humility - Keep a learning mindset
Company HQ
Mill Valley, CA
Articles
Leadership
Jason Lopatecki
(CEO)Previously a System Architect at Calix, before they moved on to found and serve as CSO/CIO at TubeMogul for over a decade.
Aparna Dhinakaran
(CPO)Has held Software Engineer roles at both Dote and Apple, and worked on Machine Learning at TubeMogul and Uber. Also co-founded and served as CEO at Monitor ML, and is additionally a columnist at Forbes.
Salary benchmarks
We don't have enough data yet to provide salary benchmarks for this role.
Submit your salary to help other candidates with crowdsourced salary estimates.
Share this job
View 17 more jobs at Arize AI