Director of AI Quality & Safety, Wolters Kluwer

• 10+ years in product quality, AI/ML systems, or related domains, with at least 3–5 years in AI-focused roles
• Demonstrated experience designing evaluation frameworks for AI/ML systems (e.g., LLM evaluation, model validation)
• Strong understanding of modern AI architectures (LLMs, RAG, agents), software quality engineering principles, and data and content validation workflows
• Experience with regulatory or compliance-heavy environments (preferred: legal, financial, healthcare)
• Proven ability to operate cross-functionally at senior levels
• Systems thinking (ability to unify software, AI, and content quality domains)
• Analytical rigor and metric-driven decision making
• Risk awareness and regulatory sensitivity
• Influence without authority in a matrixed organization
• Pragmatic execution with high standards for quality

Desirable

• The Director of AI Quality & Safety is accountable for establishing and operationalizing a comprehensive quality and safety framework across AI-enabled products, content systems, and agentic workflows within WK Legal & Regulatory
• The role ensures that AI systems are reliable, auditable, compliant, and aligned with defined quality standards, while reducing production defects and AI-related risks
• This position sits at the intersection of product, engineering, data science, and regulatory compliance, with a mandate to define measurable standards and enforce quality and product standards across the AI lifecycle
• Design and implement standardized evaluation frameworks for AI models and agentic systems (e.g., LLMs, RAG pipelines, autonomous agents)
• Define, build, and improve evaluation frameworks with SMEs for output correctness and factuality, task completion accuracy, robustness and edge-case handling
• Establish benchmark datasets and continuous evaluation pipelines (offline + online)
• Drive adoption of evaluation tooling and methodologies across product teams
• Extend the rigor of traditional software QA practices to the outputs of AI-driven systems (probabilistic outputs, non-determinism)
• Define SLAs/SLOs specific to AI performance (e.g., hallucination rate, response reliability, latency under load)
• Partner with Engineering to integrate quality gates into CI/CD pipelines
• Coordinate root-cause analysis for AI-related production issues and track implementation of systemic fixes
• Establish frameworks for validating legal and regulatory content generated or transformed by AI systems
• Collaborate with editorial and domain experts to define “ground truth” and validation protocols
• Implement human-in-the-loop and automated validation mechanisms where appropriate
• Ensure traceability between AI outputs and authoritative sources
• Define and enforce AI safety standards aligned with regulatory requirements (e.g., EU AI Act, data protection laws) and internal WK risk and compliance policies
• Implement formal standards and controls alongside internal teams for bias detection and mitigation, harmful or unsafe output prevention, and data privacy and secure handling
• Ensure auditability of AI systems (logging, explainability, decision traceability)
• Act as primary liaison with Risk, Legal, and Compliance on AI-related matters
• Define quality and safety standards and KPIs that apply across the AI lifecycle - including model and data selection, prompt and workflow design, and deployment - working in partnership with Product and Engineering
• Build dashboards and reporting mechanisms for executive visibility
• Track and benchmark performance over time and across product lines
• Track development of, and evaluate products against, external industry benchmarks and work with recognized benchmarking bodies to represent WK interests
• Drive continuous improvement loops based on measurable outcomes
• Drive QA & KPIs awareness in LR businesses and provide comms support with key findings & insights that can be used for external comms & thought leadership
• Partner with Sales, Marketing, and Customer Support on external benchmark communication and AI‑related incident response messaging
• Define operating model for AI quality & safety across CPO and DXG
• Introduce review boards, approval processes, and escalation mechanisms
• Provide guidance and enablement to product teams on quality and safety best practices
• Build and lead a small, high-impact team as the function scales
• Success Criteria:
• Clear, standardized quality and safety metrics are defined and consistently enforced across all AI-enabled products
• AI system behavior is measurable, benchmarked, and governed through repeatable frameworks
• Significant reduction in production defects and AI-related incidents
• High confidence in content correctness and traceability for legal/regulatory use cases
• Full auditability and compliance alignment for AI systems across jurisdictions