Lead Data Scientist, Dynatrace

• Advanced CS/AI/ML degree or equivalent, strong ML background
• 7+ years DS/ML, 3+ years NLP /LLMs, shipped production systems
• Python and core ML stack: 5+ years of professional Python
• Data engineering for unstructured data (3+ years): text processing, parsing, embedding-
• Friendly preprocessing
• Proven RAG expertise (1+ years): embeddings, retrieval, reranking, chunking
• Evaluation depth (1+ years): offline/online evals for accuracy, grounding, safety
• Safety/privacy (1+ years): moderation, PII/PHI redaction, policy enforcement
• LLMOps (1+ years): prompt/version management, experiment tracking, monitoring
• Excellent communication: explain trade-offs, drive data decisions

Desirable

• Our team is looking for a Lead Data Scientist specialized in Large Language Models (LLMs) to design, build, and scale generative AI capabilities for real-world, enterprise-grade use cases
• In this hands-on technical leadership role, you’ll own the end-to-end LLM stack, from data/knowledge, Ingestion and retrieval to prompt and tool-use architecture, evaluation frameworks,safety/guardrails, and cost/latency optimization
• Own the LLM system architecture: Retrieval pipelines, prompt/tool design,
• Routing/fallbacks, safety layers, and telemetry, optimized for quality, latency, and cost
• Establish technical standards for RAG: content ingestion, chunking/windowing, hybrid
• Retrieval, reranking, query understanding, and structured output contracts
• Define evaluation strategy: Create a rigorous eval suite covering answer correctness,
• Attribution/grounding, toxicity/safety, privacy leakage, determinism, latency, and cost
• Formalize LLMOps: Versioning for prompts/datasets/models, experiment governance,
• Prompt and dataset registries, and promotion criteria from dev - staging - prod
• Drive tool/agent design: API schema design for function calling, error handling, recovery
• Strategies, self-correction, and guardrail integration
• Make build-vs-buy calls: Weigh managed providers vs. open-source/self-hosted,
• Considering performance, cost, IP, privacy, and compliance
• Mentoring: Provide deep technical mentorship on prompting, retrieval design, evals, and
• Safe deployment; lead reviews of prompts, pipelines, and evaluation reports
• Hands-on Data Science
• Implement end-to-end RAG systems: ingestion - chunking - embeddings - hybrid search -
• Rerank - prompt assembly - tool calls - post-processing
• Engineer robust prompts/tools: reusable templates, multi-turn strategies, structured
• Outputs via JSON Schema/Pydantic
• Select/tune models: foundation models, embeddings, rerankers; apply LoRA/PEFT or
• Distillation when justified
• Build eval corpora: golden sets, KPIs for accuracy, groundedness, deflection, tool
• Success
• Implement guardrails: PII/PHI detection, policy prompts, jailbreak resistance, filters,
• Safety scorecards
• Productionize: ship resilient services with analytics, alerts (drift, quality, cost), SLOs, etc
• Optimize for scale: token, latency, cost; caching, context packing, batching, speculative
• Decoding, routing by intent