Fundamental - Evaluations Team Lead
Requirements
• Experience owning evaluation for tabular ML systems used in production or consequential customer decisions. • Strong statistical judgment: choosing metrics and validation schemes, estimating uncertainty, comparing models across datasets, and accounting for repeated experimentation. • Practical experience finding leakage in preprocessing, feature construction, joins, temporal dependencies, and related entities across splits. • Strong Python and SQL skills, familiarity with scikit-learn and gradient-boosted trees, and experience building reliable ML tooling or platforms used by other teams. • Experience designing fair model comparisons, including hyperparameter search, resource budgets, and end-to-end latency measurement. • Prior people management experience, including hiring, technical coaching, and performance feedback, while remaining technically involved. • Clear written and spoken communication with researchers, engineers, and customer data scientists, including the willingness to challenge claims the evidence does not support. • Experience evaluating tabular foundation models or AutoML systems. • Experience measuring how model optimisations affect predictive quality and inference performance. • Experience with relational or multi-table data, and Snowflake or Databricks environments. • Experience evaluating automated or agent-driven ML workflows, including failures that aggregate metrics can hide.
Responsibilities
• Build a shared evaluation platform for engineering, research, and Applied AI. Continuously add and maintain models and curated datasets so teams can run benchmarks and investigate results independently. • Define evaluation standards for metrics, data splits, leakage prevention, calibration, uncertainty, and benchmark contamination. • Build reproducible pipelines with versioned inputs and artifacts, and integrate regression checks into research and release workflows. • Benchmark NEXUS against competing approaches using fair tuning budgets, data access, compute, and latency measurement protocols. • Support Applied AI’s customer POC evaluations with tooling, methodological guidance, and analysis. • Measure predictive quality, latency, and cost across deployment configurations, task types, and dataset characteristics. • Turn findings into research priorities, release recommendations, and evidence-backed customer improvement plans. • Hire and develop the team, set priorities, and stay hands-on with code and experimental design.
Benefits
• Competitive compensation with salary and equity • Comprehensive health coverage for you and your dependents • Paid parental leave for all new parents, inclusive of adoptive and surrogate journeys • Relocation support for employees moving to join the team in one of our office locations • A mission-driven, low-ego culture that values diversity of thought, ownership, and bias toward action
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT