Keeper Security - Senior Software Development Test Engineer (SDET), AI & Automation
Requirements
• 5+ years of experience in SDET, software development, test automation or another code-intensive quality engineering role • Hands-on experience testing AI/ML, LLM-powered or model-integrated software systems • Strong understanding of the differences between deterministic software testing and non-deterministic AI/ML validation • Experience defining evaluation criteria, expected behaviors, tolerances or scoring approaches for AI-driven functionality • Strong programming skills in Python, Java, JavaScript/TypeScript, Rust or another general-purpose programming language • Experience building automated functional, integration, API and end-to-end test coverage • Experience testing REST APIs and distributed application workflows using tools such as Postman, REST Assured or similar platforms • Experience integrating automated tests into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI or similar technologies • Experience working within an Agile software development team with close collaboration between QA, Engineering and Product • Strong understanding of shift-left testing, continuous quality and testability principles • Experience with TestRail or similar test management platforms • Strong debugging and troubleshooting skills using logs, APIs, application telemetry and development tools • Experience with Git-based development workflows, code review and modern software engineering practices • Familiarity with AI-assisted development and testing tools such as Claude, ChatGPT, GitHub Copilot or similar technologies • Strong analytical and problem-solving skills with the ability to evaluate variable AI behavior without relying solely on exact-match outcomes • Strong written and verbal communication skills and ability to collaborate closely with AI engineers, Software Engineers, Product Managers and QA professionals • Bachelor’s degree in Computer Science, Engineering or a related technical field, or equivalent practical experience • Due to this role’s involvement with FedRAMP compliance requirements, qualified candidates must be a U.S. Person, defined as a U.S. citizen or lawful permanent resident. • Experience testing LLMs, agentic systems, retrieval-augmented generation, tool calling or prompt-driven workflows • Experience building automated model evaluation, benchmarking or regression frameworks • Familiarity with AI evaluation metrics and techniques for measuring quality across non-deterministic outputs • Experience with Rust in production or test automation environments • Experience with performance, reliability, security or adversarial testing of AI-enabled systems • Experience with AI-assisted generation of automated tests or synthetic test data • Familiarity with Docker, Kubernetes or cloud-based testing environments • Experience testing cybersecurity, identity, PAM or other security-sensitive software • Experience building shared automation frameworks or internal testing tools used across engineering teams
Responsibilities
• Partner directly with AI engineers, Software Engineers and Product Managers during design and development to define test strategy, acceptance criteria and automation requirements • Design and maintain automated testing frameworks for AI-powered applications, backend services, APIs and supporting software systems • Develop testing strategies for both deterministic software behavior and non-deterministic AI/ML outputs • Define measurable evaluation criteria, tolerances and quality thresholds for AI model and LLM-driven functionality • Build automated regression, integration, API and end-to-end coverage alongside feature development • Develop test harnesses and evaluation workflows for prompts, models, agents, tool-calling workflows and AI-integrated applications • Validate AI behavior across different models, prompts, inputs and system configurations • Develop automated test utilities using languages such as Python, Java, JavaScript/TypeScript, Rust or similar technologies • Integrate automated validation and quality gates into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI or similar platforms • Build and maintain test datasets, mocks, fixtures and synthetic data required for repeatable AI and software validation • Analyze model and application failures to distinguish defects, expected model variability, data issues and test instability • Evaluate AI outputs for reliability, consistency, safety and alignment with expected product behavior • Work with engineering teams to improve application testability, observability and diagnostic capabilities • Track test cases, execution results and coverage using TestRail or similar test management platforms • Analyze logs, telemetry and system behavior to identify root causes and produce clear, reproducible defect reports • Reduce flaky tests and improve the speed, stability and maintainability of automated test suites • Contribute to quality engineering standards, AI testing methodologies and automation best practices across Keeper’s AI organization • Use AI-assisted tools such as Claude, ChatGPT, GitHub Copilot or similar platforms to improve automation development, test generation, debugging, analysis and documentation
Benefits
• Medical, Dental & Vision (inclusive of domestic partnerships • Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life • Voluntary Short/Long Term Disability Insurance • 401K (Roth/Traditional) • A generous PTO plan that celebrates your commitment and seniority (including paid Bereavement/Jury Duty, etc)
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT