pencil - Lead Product Manager— Agent Supply & Quality - EMEA Remote
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT
Requirements
• Hands-on AI product experience — you've shipped LLM or agentic products in production. You understand evaluation, reliability, and the gap between a demo that works and an agent that works for 10,000 users. • Marketing or creative tech background — you understand the enterprise marketer's world: brand safety, content at scale, the difference between a tool that's approved and a tool that's actually used. • Strong quality instincts — you've built evaluation frameworks before. You know what a meaningful eval looks like versus a vanity metric. You're not satisfied with 'it seems to work'. • Partnership and ecosystem thinking — you've worked with external developers or partners. You understand what makes an SDK or builder compelling to build on. • Technical credibility — you don't need to write the code but you hold your own in conversations about model selection, prompt architecture, and integration design. • Rigorous and fast — you move quickly but you instrument as you go. You don't ship things you can't measure. • You'll thrive here if... • You find the gap between 'this agent works in a demo' and 'this agent works reliably for every brief a global brand throws at it' genuinely interesting to close. • You believe quality compounds — that a high-quality catalogue today is the foundation for everything the orchestration layer can do tomorrow. • You want to define what 'enterprise-grade AI agent' actually means in practice, not just in a pitch deck. • You write to think, not just to communicate. Crisp briefs and honest post-mortems are part of how you work. • KPIs & Success Measures • Eval pass rate: percentage of By Pencil core agents meeting the quality bar on automated evaluation runs against real customer briefs. • Agent coverage: number of agents live across Strategy, Creative, and Media — and MAU in non-creative personas. • Export quality: average export cost and average performance lift per export across agent-led generations. • Ecosystem growth: number of active 3P integrations live and generating usage. Agent builder adoption by external developers. • Delivery quality: P0/P1 bounce-back rate from QA. P2/P3 polish items cleared within one sprint of GA. • Shared north star: % of platform exports originating from an agent-led workflow (currently 15%).
Responsibilities
• Own the By Pencil agent roadmap — quality bar, evaluation frameworks, improvement loops, and the release criteria that determine when an agent is ready for users. • Define and scale evals — build repeatable, automated evaluation pipelines that measure agent performance against real customer briefs. Move quality from subjective to measurable. • Drive 3P integrations — own the pipeline of third-party integrations from prioritisation through to launch. Define what good integration looks like and hold the bar. • Build the ecosystem — own the agent builder experience for 1P and 3P developers. Define what makes it easy to create a high-quality agent and what the revenue share model needs to look like to attract serious partners. • Own coverage strategy — identify the gaps in agent coverage that are losing us users or deals. Build the case for what to build next and in what order. • Work backwards from the customer — before any significant build decision, write the customer problem clearly. The technology decision comes last. • Partner with engineering early — not just to hand over specs. Understand the technical constraints well enough to make good tradeoffs and give engineers clear context on why something matters. • Instrument everything — define the metrics that tell us whether quality and coverage are moving in the right direction. Set baselines, measure outcomes, feed learnings back into the roadmap. • Prototype don’t explain - build working and actionable prototypes alongside the design and engineering team to bring your ideas to life.
Benefits
• 25 days PTO plus public holidays, although we operate a Flexible Time Off scheme • Health insurance / private medical cover • Monthly stipend towards wellness, fitness, and learning and development • Remote — work from anywhere in your home country • Enhanced parental leave policies, whether you become a parent through birth, adoption or surrogacy • Access to our Pencil office in The Shard, London for our UK employees • Flexible working hours
No credit card. Takes 10 seconds.