g2 - Software Engineering Director, Agentic Evaluations
Requirements
• We realize applying for jobs can feel daunting at times. Even if you don’t check all the boxes in the job description, we encourage you to apply anyway. • 10+ years of professional programming experience in backend or full-stack environments • 2+ years of experience directly managing engineers • Expert-level proficiency in backend development using languages like Python, Java/Kotlin, Typescript/Javascript or Go; strong proficiency in associated backend frameworks like FastAPI or Node.js http://node.js • Direct experience creating evaluations (evals) against a customer-facing agent, leveraging agent trajectory trace data and rubrics to measure an agent’s task completion rate, accuracy, correctness, or policy adherence • Direct experience using frontier models from OpenAI, Anthropic, or Google in LLM-as-a-judge applications • Regular use of coding agent harnesses like Claude Code, Codex, Opencode, or Pi as part of the daily development workflow • BS/BA degree in related field • WHAT CAN HELP YOUR APPLICATION STAND OUT: • Experience with agent tool use via direct integration or as mediated by MCP servers • Knowledge of STATE-Bench, tau2-bench or similar benchmark frameworks • Experience using Playwright, browser-use, Chrome DevTools MCP or others to drive user flows in the browser • Experience with durable execution frameworks & workflow solutions, like Temporal, DBOS, Cloudflare / Vercel Workflows, or similar; or experience with agent sandboxing using AWS E2B, Daytona, Cloudflare / Vercel Containers or similar • Our Commitment to Inclusivity and Diversity • At G2, we are committed to creating an inclusive and diverse environment where people of every background can thrive and feel welcome. We consider applicants without regard to race, color, creed, religion, national origin, genetic information, gender identity or expression, sexual orientation, pregnancy, age, or marital, veteran, or physical or mental disability status. Learn more about our commitments here https://culture.g2.com/commitments. • For job applicants in California, the United Kingdom, and the European Union, please review this applicant privacy notice https://legal.g2.com/applicant-privacy-notice before applying to this job. • How We Use AI Technology in Our Hiring Process • G2 incorporates AI-powered technology to enhance our candidate evaluation process. These tools may assist with initial application screening, skills assessment analysis, and identifying candidates whose qualifications align with specific role requirements. While AI technology supports our recruitment workflow, all final hiring decisions remain under human oversight and judgment. • Your Choice Matters: If you would prefer that your application be reviewed without AI assistance, you can opt out by entering your email address in the email entry field at the bottom of the Automated Processing Legal Notice. Choosing to opt out will not disadvantage your application in any way—we will ensure your materials receive a thorough manual review by our hiring team. • For additional details about how we handle your information throughout the application process, please review G2's Applicant Privacy Notice https://legal.g2.com/applicant-privacy-notice.
Responsibilities
• Build and enhance agent evaluations that run against live agents [80%]: • Help scope out the feasibility and effort involved in evaluating an agent offered by a software vendor, identifying the most practical path to reaching a credible evaluation • Own the full stack of the core evaluation system, from admin & API surface to eval workflow dispatch • Design evaluation system primitives that generalize across software verticals, so onboarding a new category benefits from reuse • Distill the appropriate and repeatable processes in onboarding and maintaining integrations into repeatable AI skills or agents to increase evaluation velocity • Keep tabs on emerging frameworks and techniques for agent evaluation which offer improvements to evaluations, and provide industry-accepted means to distribute eval results • Collaborate with data science peers in their pursuit to develop rich and proprietary benchmarks • Lead a group of engineers and be a source of wider influence [20%]: • You will facilitate the growth and mentorship of a core team of engineers, and be a source of technical leadership and subject matter expertise in evals for them • You will also help socialize the use of evals in agent-oriented products built by the wider organization.
Benefits
• Total Earnings (Base + Bonus) $260K – $320K • Offers Equity • Offers Bonus • At G2, we’re committed to building an equitable and inclusive workplace for all. That's why our compensation program is rooted in market data, transparency, fairness, and individual performance. • Compensation in an offer is determined by factors such as the candidate’s experience, relevant skills, and job-related knowledge. • We’ve built a company-wide framework to provide employees clarity on how compensation and success is measured at each level. This career framework supports our employees' professional growth and guides them to their PEAK. • We also support our employees by offering generous benefits, such as flexible work, ample parental leave, and unlimited PTO. Click here to learn more about our benefits. • Upload your resume here to autofill key application fields. • Drop your resume here! • Parsing your resume. Autofilling key fields... • Please submit your preferred first and last name • or drag and drop here • I've designed and built an eval system for a customer-facing AI agent, live in production • I've designed and built an eval system for an internal AI agent, live in production • I've designed and built eval systems for an AI agent (customer-facing or internal), pre-launch • I've evaluated traditional ML models/pipelines, but not agentic systems • I haven't done evaluation work on AI systems • In a few sentences, describe one specific eval suite or evaluation system you personally designed and built for an AI agent. Include what the agent did, the metric(s) you measured, how you used trace/trajectory data, the tools/frameworks/LLMs involved, and roughly when (year). • Decline to self-identify • Hispanic or Latino - A person of Cuban, Mexican, Puerto Rican, South or Central American, or other Spanish culture or origin regardless of race. • Hispanic or Latino • White (Not Hispanic or Latino) - A person having origins in any of the original peoples of Europe, the Middle East, or North Africa. • White • Black or African American (Not Hispanic or Latino) - A person having origins in any of the Black racial groups of Africa. • Black or African American • Native Hawaiian or Other Pacific Islander (Not Hispanic or Latino) - A person having origins in any of the peoples of Hawaii, Guam, Samoa, or other Pacific Islands. • Native Hawaiian or Other Pacific Islander • Asian (Not Hispanic or Latino) - A person having origins in any of the original peoples of the Far East, Southeast Asia, or the Indian Subcontinent, including, for example, Cambodia, China, India, Japan, Korea, Malaysia, Pakistan, the Philippine Islands, Thailand, and Vietnam. • Asian • American Indian or Alaska Native (Not Hispanic or Latino) - A person having origins in any of the original peoples of North and South America (including Central America), and who maintain tribal affiliation or community attachment. • American Indian or Alaska Native • Two or More Races (Not Hispanic or Latino) - All persons who identify with more than one of the above five races. • Two or More Races • Hispanic or Latino • White (Not Hispanic or Latino) • Black or African American (Not Hispanic or Latino) • Native Hawaiian or Other Pacific Islander (Not Hispanic or Latino) • Asian (Not Hispanic or Latino) • American Indian or Alaska Native (Not Hispanic or Latino) • Two or More Races (Not Hispanic or Latino) • I identify as one or more of the classifications of protected veteran listed above • I am not a protected veteran
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT