Technical Project Manager, Lilt

• 3–5+ years of technical project management experience in AI/ML, data platforms, or technical data operations
• Strong understanding of LLM training and evaluation concepts, including SFT, RLHF, human evaluation, automated evaluations, LLM-as-a-judge, and red teaming
• Proficiency in SQL for analyzing delivery velocity, data quality metrics, and cost structures
• Working knowledge of data pipelines, structured data formats, APIs, and validation processes
• Experience partnering with engineers to manage technical requirements, dependencies, and issue resolution
• Proven ability to track and optimize project unit economics (cost per token/task) and delivery velocity
• Strong communication skills, including the ability to translate technical requirements for multilingual audiences and diverse global contributor networks
• Experience managing complex workflows using Agile, Scrum, or Kanban

Desirable

• We are looking for a Technical Project Manager to join our Applied AI team and lead large-scale multilingual data collection, human-in-the-loop workflows, and Large Language Model (LLM) evaluation initiatives
• You will work with AI specialists, engineering teams, specialized Subject Matter Experts (SMEs), and data operations teams to translate technical AI requirements into high-throughput data workflows, hybrid evaluation pipelines, and measurable delivery plans
• This role combines technical project management with hands-on data analysis, unit economics optimization, and a strong operational understanding of modern LLM training and evaluation processes
• Manage Applied AI data collection and evaluation projects from technical scoping through validation, delivery, and retrospective review. through implementation, validation, delivery, and retrospective review
• Translate technical AI requirements into structured data specifications, domain-specific annotation requirements, evaluation plans, and acceptance criteria
• Manage timelines and throughput velocity, dependencies, delivery risks, and technical blockers across Applied AI, engineering, and operations stakeholders
• Coordinate data collection and annotation for supervised fine-tuning (SFT), preference data used in reinforcement learning from human feedback (RLHF), and model evaluation
• Partner with engineering teams to define and validate data workflows, including ingestion, annotation tooling, systems integrations, quality checks, data generation pipelines, and delivery
• Operationalize hybrid evaluation methodologies, combining human review with automated evaluation tooling, and LLM-as-a-judge frameworks
• Coordinate requirements for data formats, schemas, metadata, annotation tools, and system integrations
• Coordinate requirements for data formats, schemas, metadata, annotation tooling, and automated quality checks
• Work with Applied AI specialists to operationalize evaluation methods, including human review, response ranking, and safety testing
• Investigate data, pipeline, or tooling issues and coordinate prompt resolution with technical owners
• Use SQL and business intelligence tools to analyze throughput velocity, quality, unit economics, and supplier/delivery performance
• Define and monitor quality standards, including inter-annotator (IAA) agreement, gold-set performance, and dataset completeness
• Lead root-cause analysis on quality discrepancies, workflow bottlenecks, benchmark discrepancies, and quality edge cases
• Establish delivery validation checks and coordinate stakeholder acceptance of completed, accurate datasets, delivery targets, and evaluation results
• Translate complex technical requirements into actionable plans and clear instructions for global contributor teams, external vendors, and specialized domain SMEs and teams
• Communicate project status, technical risks and tradeoffs, unit economics, and operational risks to technical and non-technical stakeholders
• Ensure contributor feedback and evaluation findings inform improvements to internal tooling, guidelines, and dataset pipelines