Senior Data Scientist, Alpaca

• Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation)
• Strong statistical and ML foundation—you treat evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validate automated graders against human ground truth
• Proficiency in Python and SQL, with experience evaluating models in production environments
• Strong judgment in defining quality metrics and ground truth for ambiguous outputs
• Excellent communication and cross-functional collaboration skills to align technical teams and leadership
• Strong problem-solving ability in fast-paced, greenfield environments
• 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation. A quantitative degree is a plus; equivalent industry experience is equally welcome

Desirable

• Design AI evaluations: Define ground truth, metrics, and scoring methods for models and agents
• Build repeatable eval loops: Track quality over time and catch regressions before release
• Partner on infrastructure: Work with engineering and analytics engineering to operationalize eval harnesses
• Drive iteration: Translate eval results into actionable recommendations for system improvements
• Establish quality standards: Set evaluation guidelines, documentation, and review practices
• Mentor and align: Foster evaluation best practices and build a culture of measurable AI quality across the team