DEFCON AI - Model Test and Measurement Engineer

Remote - USA$150k - $190k+ Equity3w ago

Remote Senior NA Government Test Engineer Python SQL Documentation

Requirements

• 5+ years with model validation as a named responsibility, not a side task • Direct experience with ground-truth construction and sampling design • Strong Python and SQL; comfort working independently from the teams whose models you evaluate • US Citizenship Required • Active US Secret clearance • Elevated personnel security requirements apply to portions of this work and are discussed during screening • Regulated-industry or government model-risk background • Experience with fairness / disparate-effect testing and human-oversight documentation • NIST AI RMF or comparable practice • Active Top Secret clearance • What Success Looks Like • Evaluation evidence a customer can rely on, independent of the teams that build the models • Releases gated against a clear version and evaluation record • Workflow metrics (review time, throughput, backlog, override/rework rates) that give the program an honest read on whether it's working

Responsibilities

• Construct labeled ground truth for model evaluation • Design and run sampled audits • Gate releases against model version; maintain version inventory, evaluation records, and rollback triggers • Run drift and override review • Produce human-oversight and fairness / disparate-effect evidence • Maintain independence from the build roles: produce evaluation evidence, but do not build the models, approve thresholds, or authorize releases against that evidence • Measure workflow improvement: review time, throughput, backlog movement, and override and rework rates • Design evaluation-phase QC: sample selection that does not mix populations, the unit of review, and fair comparison when methods or searched sources differ • Define the measurement events other teams must emit, and confirm the review workspace emits workflow events from first use

Benefits

• A fully remote, results-based environment • Competitive salary, bonus, and equity package • 100% employer paid, comprehensive health insurance including medical, dental, and vision for you and your family • Unlimited PTO, with your manager's approval • Flexible work environment where you manage your work day • 14 weeks of fully-paid parental leave • Salary Range: $150,000–$190,000. This represents the typical salary range for this position based on experience, skills, and other factors.