Senior Quality Assurance Analyst, Keeper Security

• This role requires hands-on Python automation expertise, strong analytical and evaluation skills, and the ability to test nondeterministic systems with rigor and consistency
• You will collaborate closely with ML engineers, security teams and product engineering to ensure AI features meet Keeper’s quality, safety and performance standards
• 3+ years of experience in QA, test automation or evaluation of complex systems
• Hands-on experience writing Python automation or evaluation scripts for testing APIs, model outputs or data pipelines
• Understanding of machine learning concepts such as classification, summarization, embeddings, anomaly detection or behavior analysis
• Experience testing LLM-integrated features or AI-driven workflows
• Familiarity with ML tooling such as Pytest, Hugging Face libraries, Jupyter Notebooks or similar
• Experience with API testing, JSON validation and test data generation
• Ability to test nondeterministic systems and evaluate model quality with structured methodologies
• Strong analytical, troubleshooting and communication skills
• Experience testing cloud-integrated applications or systems deployed on AWS
• Bachelor’s degree in Computer Science, Information Systems or related field, or equivalent practical experience

Desirable

• The Senior QA Analyst AI & Threat Analytics will lead testing efforts for AI-driven features and machine learning models across Keeper’s platform, including LLM-based summarization, autofill and classification systems, and behavior analysis pipelines
• You will design and execute test strategies that validate model accuracy, reliability, guardrails, data quality, and integration with production systems
• Design and execute test plans for AI and ML-driven features, including classification, summarization, anomaly detection and autofill systems
• Develop Python-based automation and evaluation scripts to validate model performance, reliability and output quality
• Test LLM-based features for accuracy, guardrail effectiveness, hallucination risk and behavior under edge cases
• Validate data pipelines, input/output transformations and model integration points across Keeper’s platform
• Conduct regression testing on ML models and AI workflows to ensure reliability across releases
• Test enterprise-scale behavior analysis features driven by data from ARAM, PAM and session recordings
• Validate security, privacy and zero-trust requirements for AI model interactions and data handling
• Collaborate with ML engineers and cross-functional teams to refine model behavior, identify defects and improve output quality
• Analyze logs, metrics and model outputs to detect anomalies, performance issues or silent failures