Wizard - Data Scientist - AI Evaluation

• Define and evolve accuracy metrics across the full shopping experience (retrieval, ranking, recommendations and outcomes) • Design and run experiments to measure improvements and regressions • Build and maintain evaluation datasets, benchmarks and scoring frameworks • Translate ambiguous product questions into clear, measurable hypotheses and analysis • Partner with ML Engineers to validate model changes and guide iteration • Identify failure modes and edge cases and drive improvements through data • Create dashboards and reporting that make agent performance visible, trusted and actionable • What Success Looks like • Clear, trusted accuracy metrics are consistently used across product and engineering • A robust automated evaluation framework exists for both offline and live experiments • Model and product changes are consistently measured before and after launch • Ideal Background • Ideal Background • Deep experience evaluating AI/ML systems (ranking, recommendations, LLMs, etc) • Strong experience with experimentation (A/B testing, causal inference) • Experience working on consumer products or user facing systems and exposure to marketplace or e-commerce systems • Ability to translate messy problems into structured analysis and metrics • Strong product mindset, you care about real user outcomes • Clear communication with the ability to influence across engineering and product