Senior QA Automation Engineer, Dow Jones

• Proven experience as a Senior QA Automation Engineer / SDET in production environments
• Strong hands-on experience with Playwright (or equivalent) using TypeScript
• Demonstrated use of AI coding tools (e.g. Claude Code) as part of daily workflow
• Strong software engineering fundamentals and ability to write maintainable production-grade code
• Experience designing automation frameworks and test strategies end-to-end
• Deep understanding of CI/CD and integrating automation into delivery pipelines
• Strong debugging and observability skills across distributed systems
• Experience driving shift-left practices and enabling engineers to own quality
• Solid understanding of databases, APIs, and test data validation (SQL, REST/GraphQL)
• Security-aware mindset (OWASP, secure test design)
• Familiarity with LLM observability and experimentation tools (e.g. Langfuse, MLflow)

Desirable

• Storyful is building AI-native products on top of complex, high-volume, multi-source content. We are looking for a hands-on Senior QA Automation Engineer to raise the quality bar while increasing delivery speed through AI-augmented engineering
• This is a senior individual contributor role for someone who treats AI as a core part of their craft — using coding agents, LLMs, and evaluation frameworks to scale quality faster than headcount
• You will design and build the automation, evaluation, and quality foundations that enable engineers to own quality, while ensuring our products remain reliable, explainable, and trusted — even as they become increasingly probabilistic
• You will work closely with software engineers, data scientists, product managers, and SRE to embed quality into every stage of the development lifecycle
• Build and evolve scalable automation frameworks across web UI, APIs, and data pipelines (Playwright / TypeScript)
• Use AI coding agents (e.g. Claude Code) to generate, refactor, and maintain test suites at scale
• Translate natural-language acceptance criteria into executable, maintainable tests
• Define reusable testing patterns that enable engineers to own automation within their domains
• Continuously reduce flakiness using trace analysis, observability, and AI-assisted failure triage
• Partner with engineers to embed testing early in the development lifecycle
• Define pragmatic, risk-based test strategies across unit, integration, contract, and E2E layers
• Establish clear definitions of done, quality gates, and release readiness standards
• Build tooling and frameworks that allow teams to move fast without relying on central QA
• Design evaluation pipelines for LLM-driven features using:
• Golden datasets and regression harnesses
• Semantic similarity scoring
• LLM-as-a-judge evaluation patterns
• Detect and prevent issues such as prompt drift, hallucinations, retrieval failures, and schema regressions
• Test agentic workflows end-to-end (tool use, multi-step reasoning, failure modes)
• Partner with data science to define measurable quality signals (precision, recall, grounding, drift)
• Embed automation and evaluation into CI/CD pipelines for fast, reliable feedback
• Define and tune meaningful quality gates based on real metrics (not ceremony)
• Support release processes with high-confidence signals and rollback readiness
• Investigate complex issues across distributed systems using logs, traces, and metrics
• Improve observability, including LLM trace visibility (e.g. Langfuse)
• Establish strong defect triage, categorisation, and reporting practices
• Use data (leakage, flake rate, MTTR, escape rate) to continuously improve quality
• Embed security testing into automation (OWASP, API security, auth flows)
• Test AI-specific risks (prompt injection, jailbreaks, unsafe tool use, data leakage)
• Collaborate on performance, resilience, and scalability testing where it matters most
• What good looks like in this role:
• Engineers own quality by default, supported by strong frameworks and AI tooling
• Automation is fast, reliable, and trusted as a release signal
• LLM-powered features ship with measurable, repeatable quality evaluation
• Regression risk is proactively managed across both deterministic and probabilistic systems
• Delivery speed increases without compromising trust or stability
• Quality becomes a competitive advantage, not a bottleneck