Prolific - Human Data Quality Engineer (Founding Team)
Requirements
• Experience with LLM evaluation, RLHF, AI safety or red teaming. • Experience translating vague model or evaluation goals into clear annotation specifications. • Experience working with human annotation programmes or human data operations. • Familiarity with calibration, inter-rater agreement, drift detection or other evaluation methodologies.
Responsibilities
• Design the quality frameworks that underpin complex human data programmes, from evaluation rubrics through to launch readiness. • Work with clients as a strategic thought partner, challenging annotation schemas and data requirements when they won't produce the signal the model needs. • Advise on project design and how the choice of schema can impact data quality. • Build quality upstream across the operational workflow, from recruitment, screening, and training through to writing guidelines and running calibration sessions. • Build scalable quality systems, measurement frameworks and automated checks using Python and SQL. • Partner with product and engineering to build the quality infrastructure that delivers high-quality human data at scale. • Investigate data quality and integrity issues, identifying root causes and turning insights into scalable improvements. • Architect and build dashboards, monitoring and reporting that provide clear visibility into quality and operational performance. • Raise the quality capability across the company, upskilling operations and acting as a thought mentor to junior analysts. • Help define how Prolific approaches quality across new AI domains, shaping best practice as the team grows.
Benefits
• We've built a unique platform that connects researchers and companies with a global pool of participants, enabling the collection of high-quality, ethically sourced human behavioral data and feedback. This data is the cornerstone of developing more accurate, nuanced, and aligned AI systems. • We believe that the next leap in AI capabilities won't come solely from scaling existing models, but from integrating diverse human perspectives and behaviors into AI development. By providing this crucial human data infrastructure, Prolific is positioning itself at the forefront of the next wave of AI innovation – one that reflects the breath and the best of humanity. • Working for us will place you at the forefront of AI innovation, providing access to our unique human data platform and opportunities for groundbreaking research. • Links to More Information • External Handbook • By submitting your application, you agree that Prolific may collect your personal data for recruiting and global organisation planning. Prolific's Candidate Privacy Notice explains what personal information Prolific may process, where Prolific may process your personal information, its purposes for processing your personal information, and the rights you can exercise over Prolific use of your personal information.
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT