JetBrains - Senior Machine Learning Engineer (IntelliJ AI)

• Design and drive evaluation frameworks for AI features, including metrics, experiments, and agent trace analysis. • Diagnose model performance issues (e.g. prompt drift, context mismatches, and latency/quality trade-offs) and translate findings into actionable improvements. • Experiment with contexts and lightweight models to continuously develop our ML system. • Act as the ML liaison for product teams across JetBrains, adapting and scaling AI capabilities in JetBrains IDEs to their needs. • Build and maintain small helper models (e.g. re-rankers, classifiers, embedding models) to support domain-specific tasks. • Collaborate with colleagues in ML, product, engineering, and analytic teams to deliver improvements and monitor their impact in production. • Stay up to date with research in the fields of LLMs, agents, and evaluation, bringing best practices into our workflows. • Mentor junior engineers and help shape team culture, processes, and tooling around experimentation and evaluation. • We’d be happy to have you on our team if you: • Have 5+ years of experience as an ML Engineer, with a solid background in production-grade ML systems (especially LLMs and agent architectures). • Have experience with LLM evaluation methods and frameworks. • Can design and run end-to-end experiments – hypotheses, metrics, data collection (including traces/logs), analysis, and decision-making. • Are skilled in context-aware pipelines or conversational/agent systems. • Have strong Python programming skills. • Bring hands-on experience in fine-tuning or training smaller models (e.g. domain-specific fine-tuning and lightweight customizations). • Communicate clearly and effectively across teams, translating ML/AI insights into product features. • Have prior mentorship experience with ML/evaluation engineers. • Thrive in a cross-functional, fast-moving environment, taking ownership, iterating quickly, and delivering results. • We’d be especially thrilled if you have: • Familiarity with agent-based systems and orchestrating multi-step reasoning agents. • Experience with the Kotlin programming language.