Senior Data Scientist, EvolutionIQ

• 5+ years of experience in Data Science with a strong background in traditional statistics (hypothesis testing, experimental design, regression analysis)
• 2+ years of focused experience working with LLMs, specifically in evaluation, benchmarking, and prompt auditing
• Master’s or PhD in Statistics, Mathematics, or a related quantitative field. You should be comfortable explaining the nuances of different evaluation metrics (e.g., G-Eval, ROUGE, or custom-weighted scorecards)
• A Quality Mindset. You are naturally skeptical of anecdotal evidence ("it looks good") and insist on statistical proof
• Proven ability to work with non-technical SMEs to translate their qualitative feedback into quantitative metrics
• Proficient in Python (Pandas, Scikit-learn, Statsmodels) and SQL. Familiarity with LLM evaluation frameworks (e.g., RAGAS, LangSmith, or proprietary scorecard systems) is a major plus

Desirable

• As a Senior Data Scientist on the Medhub team, you will be the primary architect of our LLM evaluation framework. Think of this role as the "AI equivalent of a Quality Engineer"—you aren't here to build the models, but to define the rigorous statistical standards they must meet before they ever touch production
• You believe that LLMs should be held to the same (or higher) scientific standards as traditional supervised models
• You enjoy the challenge of turning subjective outputs (like chat and summarization) into objective, measurable data. You are a "Data Scientist’s Data Scientist"—someone who leans heavily into statistical significance, inter-rater reliability, and robust experimental design to ensure our AI products are safe, accurate, and reliable
• Establish the Gold Standard: Design and implement comprehensive scorecards and benchmarking suites for LLM-based extraction, summarization, and chat interfaces
• Bridge the Gap with SMEs: Act as the technical lead in working with Subject Matter Experts (SMEs) to codify their expertise into evaluation datasets and "ground truth" labels
• Scale Labeling: Design the statistical guardrails to scale both our human and automated labeling efforts; you will optimize the human labeling process for efficiency and agreement while ensuring LLM-generated labels maintain the highest integrity for our training data
• Quantify Risk: Provide clear, data-driven "Go/No-Go" recommendations for model deployment based on rigorous error analysis and statistical confidence intervals