Caseware - AI Test Architect

• Maira Russo - Senior Talent Acquisition Partner • 1.AI-Driven Quality Strategy & Architecture • Architect a comprehensive "Quality Intelligence" platform using generative AI to predict defect hotspots, intelligently optimize regression suites, auto-generate tests, and enable self-healing automation. • Define enterprise-wide AI-first testing strategy, including non-deterministic evaluation paradigms, continuous monitoring for drift/hallucination, and integration across the full SDLC. • Establish governance for ethical AI testing, aligning with emerging standards • 2.LLM & Agent Evaluation Frameworks • Design and implement advanced benchmarks, red teaming protocols, and adversarial testing for internal AI agents and generative features—focusing on hallucination rates, bias/fairness, prompt injection, jailbreaks, and goal misalignment. • Build evaluation pipelines with statistical rigor (e.g., multi-trial runs, LLM-as-judge, human-in-the-loop) using tools like LangFuse, LangSmith, DeepEval, RAGAS, or Arize Phoenix for metrics such as faithfulness, context precision, and safety compliance. • Architect harnesses for agentic workflows, tool-calling, planning, multi-agent simulations, and post-deployment observability. • 3.Infrastructure & Automation Architecture • Partner with DevOps to embed AI-based testing into GitHub-based CI/CD pipelines (e.g., AI-generated tests, predictive flakiness detection, automated gating with quality signals). • Lead design of self-healing test frameworks (integrating AI plugins with Playwright/Cypress or similar) that adapt to UI/model changes with minimal maintenance. • Architect synthetic data generation, maintain golden data-sets and AI-powered data masking solutions to enable privacy-compliant, high-fidelity testing at scale. • 4.Cross-Functional Leadership & Evangelism • Collaborate with product, data science, ML engineering, and security teams to influence AI feature design with quality guardrails from day one. • Evangelize and mentor: Upskill traditional QA engineers into AI-augmented testers through workshops, playbooks, and communities of practice. • Drive adoption of AI quality best practices organization-wide, including metrics dashboards for DORA + AI-specific indicators (e.g., hallucination rate, red team success rate, self-healing coverage). • 5.Observability, Metrics & Continuous Evolution • Define and implement AI-specific quality telemetry (e.g., drift detection, faithfulness scoring, compliance incidents) integrated with tools like Langfuse. • Establish feedback loops for model iteration, A/B testing guardrails, and proactive risk mitigation in production. • Challenges You'll Architect Solutions For • Building reliable evaluation for non-deterministic, agentic AI in a fast-moving SaaS landscape. • Scaling self-healing and generative test automation without introducing new flakiness or security debt. • Balancing innovation speed with rigorous red teaming and ethical safeguards for customer-facing AI. • Success in the First 6-12 Months • Launch the "Quality Intelligence" platform foundation with AI-augmented pipelines covering > 70%+ of critical paths. • Establish red teaming/red-teaming-as-code processes that reduce high-severity AI risks by > 40%+. • Upskill > 50%+ of QA/engineering teams on AI testing fundamentals and deliver measurable velocity/safety gains. • Accuracy Baseline: Establish a baseline 90%+ Faithfulness score for all RAG-powered features. • 8+ years in Quality Engineering/Test Architecture within cloud-native SaaS environments, with 2+ years focused on AI/ML/LLM testing and validation. • Deep expertise in AWS (serverless, microservices, IaC with Terraform/CloudFormation) and GitHub CI/CD ecosystems. • Proficiency architecting LLM-based applications and testing frameworks (LangChain/LangGraph/LangSmith strongly preferred; equivalents acceptable). • Mastery of modern automation (Playwright, Cypress) with hands-on experience integrating self-healing AI plugins or generative test tools. • Strong programming skills in JavaScript/TypeScript and/or Python; solid understanding of foundational AI concepts (transformers, embeddings, RAG, evaluation trade-offs). • Experience with LLM evaluation tools like Bedrock Evaluations, Prompt Management, Guardrails, DeepEval, RAGAS, Arize Phoenix, Langfuse. • Experience with Red teaming frameworks/tools (Cobalt Strike, Sliver, Nmap) and knowledge of adversarial testing methodologies is a bonus. • Proven leadership: Mentoring teams, defining standards, and driving cross-functional change in ambiguous, high-growth settings. • Bachelor's/Master's in Computer Science, AI/ML, or equivalent; relevant certifications a strong plus. • Strong English language communication and collaboration skills