Nous Research - Machine Learning Engineer, Evals

• 3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job • Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short • Hands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents • Solid Python. You write clean, tested, version-controlled code that a colleague could run without you babysitting it • Comfort with Git, CI/CD basics, Docker, and the Linux command line (SSH, tmux, debugging a remote job) • Understanding of basic eval statistics: why accuracy misleads on imbalanced judges, what Cohen's κ measures, how to think about confidence intervals on a metric • At least 3 of the following: you can explain why LLM-as-judge needs calibration; you've done failure analysis and can tell model bugs apart from prompt, grader, or retrieval issues; you know at least two agent benchmarks (GAIA, AgentBench, τ-Bench, MINT, SWE-bench, WebShop, ALFWorld) and a limitation of each; you've designed or extended an eval dataset with happy paths, edge cases, and adversarial examples; you've thought about non-determinism in eval, how you sample, how many runs, how you report variance • You communicate clearly to both researchers and engineers, in the right language for each • You're comfortable with ambiguity, can turn a half-formed request into a plan, and know when to ask for help • Preferred • Distributed eval orchestration experience • Benchmark design from scratch • Red teaming and adversarial eval experience • Familiarity with psychometrics or measurement theory