Cresta - AI Evaluations Lead
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT
Requirements
• 5+ years of experience in Quality Engineering, Deployments, or Technical QA, ideally within an AI or high-growth SaaS environment. • Systems Thinking: A strong technical intuition and curiosity about how LLMs work. While you don't need to code, you must be comfortable navigating technical concepts like LLM, RAG, prompt logic, and multi-turn conversational flows. • Systems Thinking: • Operational Leadership: Proven ability to large E2E technical projects through partners, and a passion for building processes that improve efficiency between QA, Engineering, and Product. • Operational Leadership: • The "QA Nose": An uncanny ability to find the edge case and a bias toward action. You anticipate bottlenecks before they happen and deliver solutions with urgency. • The "QA Nose": • High Empathy: A consultative mindset with the ability to represent the "human element" of a customer support interaction. • High Empathy: • Startup Agility: You thrive in fast-paced environments, excel at turning ambiguity into execution, and are comfortable "rolling up your sleeves" to build. • Startup Agility: • Experience with CCaaS (Contact Center as a Service), telephony, or STT/TTS (Speech-to-Text) technologies. • Background in Conversation Design or SDET roles. • Experience leading team with direct reports.
Responsibilities
• Architect & Scale AI Evaluation Systems: Design and oversee the end-to-end framework for testing AI agent systems at scale. You will leverage LLM-driven methodologies—including automated simulations, "LLM-on-LLM" rubrics, and adversarial red-teaming—to ensure reliability, policy adherence, and logic across complex, multi-turn conversational flows. • Architect & Scale AI Evaluation Systems: • Drive Deployment Excellence: Partner with Forward Deployed Engineers and PMs to triage issues, identify bottlenecks, and create new test cases on the fly to address real-world deployment challenges. • Drive Deployment Excellence: • Be the Customer’s Voice: Conduct manual UAT and voice-call testing to represent the end-customer experience. You take it personally when an agent lacks empathy or clarity, and you excel at articulating these nuances to the engineering team and clients. • Be the Customer’s Voice: • Lead and Scale the Team: lead a pod of QA analysts and partners. You will define the best practices, communication loops, and shared knowledge base that allow the QA function to scale alongside our rapidly growing product line. • Lead and Scale the Team:
Benefits
• We offer a comprehensive and people-first benefits package to support you at work and in life: • Comprehensive medical, dental, and vision coverage with plans to fit you and your family • Flexible PTO to take the time you need, when you need it • Paid parental leave for all new parents welcoming a new child • Retirement savings plan to help you plan for the future • Remote work setup budget to help you create a productive home office • Monthly wellness and communication stipend to keep you connected and balanced • In-office meal program and commuter benefits provided for onsite employees • Cresta’s approach to compensation is simple: recognize impact, reward excellence, and invest in our people. We offer competitive, location-based pay that reflects the market and what each individual brings to the table. • Compensation for this position includes a Base salary + Bonus + Equity. • Actual base salaries will be based on candidate-specific factors, including experience, skillset, and location, and local minimum pay requirements as applicable. Your recruiter can provide further details. In addition, total compensation includes a comprehensive benefits package for you and your family. • We have noticed a rise in recruiting impersonations across the industry, where scammers attempt to access candidates' personal and financial information through fake interviews and offers. All Cresta recruiting email communications will always come from the @cresta.ai domain. Any outreach claiming to be from Cresta via other sources should be ignored. If you are uncertain whether you have been contacted by an official Cresta employee, reach out to [email protected]
No credit card. Takes 10 seconds.