Tech Lead/Manager, Scale AI
Machine Learning Research Scientist- LLM Evals
$240-380k
+ Equity
1-2 days a week in office
Training and validation data for AI applications
Job no longer available
Training and validation data for AI applications
1001+ employees
Job no longer available
$240-380k
+ Equity
1-2 days a week in office
1001+ employees
Company mission
To develop reliable AI systems for the world's most important decisions.
Role
Who you are
- 5+ years of hands-on experience in large language model, NLP, and Transformer modeling, in the setting of both research and engineering development
- Experience and track of recording in landing major research impacts in a fast-paced environment
- Experience supporting and leading a team of research scientists and research engineers
- Excellent written and verbal communication skills
- Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals
- Previous experience in a customer facing role
What the job involves
- Our Research teams work with the industry’s leading AI labs to provide high quality data and accelerate progress in GenAI research
- As the Tech Lead Manager of the LLM Evals Research team, you will lead a talented team of research scientists and research engineers focused on developing and implementing novel evaluation methodologies, metrics, and benchmarks to assess the capabilities and limitations of our cutting-edge LLM
- This role is critical for designing and executing a roadmap that defines best practices in data driven AI development and will accelerate the next generation of generative AI models in partnership with top foundational model labs
- Lead a team of highly effective research scientists and research engineers on LLM evals
- Conduct research on the effectiveness and limitations of existing LLM evaluation techniques
- Design and develop novel evaluation benchmarks for large language models, covering areas such as instruction following, factuality, robustness, and fairness
- Communicate, collaborate, and build relationships with clients and peer teams to facilitate cross-functional projects
- Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols
- Implement scalable and reproducible evaluation pipelines using modern ML frameworks
- Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives
- Remain up-to-date on ongoing research in the team, help work through technical challenges, and be involved in design decisions
- Remain deeply involved in the research community, both understanding trends, and setting them
- Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results
Our take
When training AI models, lack of access to large scale, high quality, labelled data is often the barrier preventing machine learning teams from building something great. In fact, labelled data is even more essential than the algorithms themselves. Scale AI is solving this by providing easy access through developer APIs to a huge variety of labelled data sets covering a large range of areas within computer vision and natural language processing.
It seems that almost every company now is offering artificial intelligence enabled products to automate every aspect of our lives. So, by positioning themselves as the go-to source for high quality tagged data, Scale AI is in an excellent position as the AI sector continues it's exponential growth.
With innovative AI projects across a range of sectors including retail, energy, and healthcare, Scale AI is now a leader in the industry alongside OpenAI. It continues to forge partnerships with fintech tools and software engine developers such as Brex and Luminar to continue its evolution of digital markets.
Freddie
Company Specialist at Welcome to the Jungle
Insights
Company
Funding (last 2 of 7 rounds)
May 2024
$1bn
SERIES F
Apr 2021
$325m
SERIES E
Total funding: $1.6bn
Company benefits
- Health & Wellbeing: Our holistic approach to supporting Scaliens includes comprehensive health coverage, dental and vision insurance, mental healthcare services, and more. PTO policies and accommodating schedules ensure you’ll get time off when you need it to relax and recharge. Note that our offerings may vary by region as we strive to respond to the unique needs of Scaliens around the globe.
- Personal & Career Growth: Continuously learn and grow through annual learning & development stipend, attending leadership breakfasts, manager training, speaker series, and joining an ERG.
- Building Scale Community: We welcome guests to our offices, and you can expect to see Scalien families and friends around. Join local happy hours, and accept invites to game nights, book clubs, and many other employee-led community events.
- Parental Support: Balancing work and family is essential, and Scale understands the importance of having adequate leave policies in place to promote a healthy home and work life.
Company HQ
Rincon Hill, San Francisco, CA
Leadership
Alexandr Wang
(Founder & CEO)Previously, Alexandr was an Algorithm Developer at Hudson River Trading, a Tech Lead and Quora and a Software Engineer at Addepar.
Salary benchmarks
We don't have enough data yet to provide salary benchmarks for this role.
Submit your salary to help other candidates with crowdsourced salary estimates.
Share this job
View 100 more jobs at Scale AI