Senior Platform Engineer, Khan Academy
AI Evaluation, 24 Months Fixed-Term
$137.9-172.3k
Canada rates: $186,306 - $232,883 (CAD)
More information about location
Nonprofit online education platform
Be an early applicant
Nonprofit online education platform
1001+ employees
Be an early applicant
$137.9-172.3k
Canada rates: $186,306 - $232,883 (CAD)
More information about location
1001+ employees
Company mission
Khan Academy's mission is to provide a free, world‑class education for anyone, anywhere.
Role
Who you are
- This is largely a software development role, but domain experience with AI eval is essential for appreciating the hill-climbing and data science workflows we need to support
- Soft skills will be important for gathering internal requirements, getting buy-in for changes, and then developing documentation and training materials
- Be fluent in the range of offline and online evaluation strategies, and when to apply the techniques over the lifecycle of development
- Have intuitions about how to specify eval pipelines succinctly using declarative syntax
- Understand the role of stratified datasets and ground truth labeling
- Appreciate the range of eval scoring schemes from human raters to automated LLMs-as-judge
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field
- 5 years of Software Engineering experience with 2+ of those years working on the evaluation of generative AI systems
- Strong programming skills in Go, Python, SQL, and at least one data pipeline framework (e.g., Airflow, Dagster, Prefect)
- Familiarity with the architecture of large language models and their industry-standard APIs
Desirable
- Experience with labeling platforms (e.g., Label Studio, Scale AI, Toloka) and human-in-the-loop concerns such as rubric development and inter-rater agreement
- Exposure to MLOps practices such as model registry, feature store, or continuous evaluation
- Background in education technology or other human-centered AI applications
What the job involves
- We’re looking for an AI Platform Engineer to evolve and extend our internal evaluation framework for assessing the quality of our AI-driven experiences at Khan Academy
- This engineer will have worked with enough eval systems to quickly make sense of Khan's internal eval framework and recognize opportunities for improvement
- You’ll work closely with ML data engineers and platform developers to help internal teams adopt an eval-driven development process incorporating offline benchmark tests and online experiments
- Be fluent in the range of offline and online evaluation strategies, and when to apply the techniques over the lifecycle of development
- Have intuitions about how to specify eval pipelines succinctly using declarative syntax
- Understand the role of stratified datasets and ground truth labeling
- Appreciate the range of eval scoring schemes from human raters to automated LLMs-as-judge
Our take
Khan Academy is a non-profit that provides free online education materials, including teacher tools, instruction videos, and practice exercises to support users of all ages. It has over 10 million users a month and covers subjects all the way up to college level.
With nearly a quarter of American adults unable to use mathematics in daily life, Khan Academy’s interactive quizzes, instructional videos, and teacher tools are timely. Khan Academy started life as a series of educational YouTube videos, before building a full-scale platform after receiving an influx of donations from Silicon Valley philanthropists. This background has led it to embrace rather than reject the educational possibilities of new tech such as AI, with the founder regarding traditional institution's banning of tools like ChatGPT as a mistake.
The company offers its tools completely for free, instead relying on donations and volunteers. A key focus is tackling wealth inequality in education. A partnership with the College Board (who produce entrance examinations for higher education) has made access to SAT test prep materials free for millions. 2024 saw a partnership with Microsoft for Khan Academy's Khanmigo product, now free to use for Microsoft customers.
Steph
Company Specialist at Welcome to the Jungle
Insights
23% employee growth in 12 months
Company
Funding (last 2 of 5 rounds)
Jul 2020
$3m
GRANT
Oct 2017
$3m
GRANT
Total funding: $16.2m
Company benefits
- Flexible spending account (FSA)
- Dental insurance
- Optical insurance
- Life insurance
- Performance bonus
- 401K plan
- Family medical leave
- Diversity program
- Pet friendly workplace
- Work from home opportunities
- Health insurance
- Affinity groups where Parents, Black, Hispanic/Latino, Women and Gender Minorities, and LGBTQIA+ identified folks come together
Company HQ
Mountain View, CA
Leadership
Caroline Hu Flexer
(CEO)Holds a BA and MArch in Architecture from Stanford and Harvard University. They also have an MBA from Stanford Graduate School of Business, and formerly co-founded Duck Duck Moose, a mobile educational platform for children.
Salman Khan
(Executive Director)They graduated from Harvard Business School and have previously worked in Finance.
Salary benchmarks
We don't have enough data yet to provide salary benchmarks for this role.
Submit your salary to help other candidates with crowdsourced salary estimates.
Share this job
View 13 more jobs at Khan Academy