HighLevel - Quality Analyst II
Requirements
• Insatiable curiosity and a self-learning mindset — this is non-negotiable. The AI space changes weekly and we hire people who level up on their own without waiting for permission or perfect instructions. • 2+ years of experience in Quality Assurance, Customer Support, AI evaluation or a related role where you reviewed and improved customer-facing or advisor-facing work • Working understanding of how LLMs behave — prompt design, common failure modes, conversational AI patterns • General understanding of databases and how data flows between systems • Comfort reading code and making minor edits — enough to follow integrations, debug prompt behavior, iterate inside ORA and hold informed technical conversations with engineering. You do not need to be a full-stack engineer. • Strong written communication and attention to detail — you can document a failure mode so clearly that an engineer can act on it without follow-up • Ability to follow structured guidelines and scoring rubrics consistently while suggesting improvements to them • Comfort working with evolving tools and AI-assisted workflows • Self-starter mentality — you can manage review volume, prioritize what matters and move work forward with minimal direction • Bachelor’s degree in a related field, or equivalent practical experience • Hands-on experience with AI development platforms such as Claude, Cursor or similar tools • Familiarity with retrieval-augmented generation (RAG), vector databases or knowledge-base architecture • Familiarity with SQL or similar database querying • Experience with APIs, webhooks or data integration between systems • Background in customer success, support, sales enablement or call center environments • Experience with the HighLevel platform or comparable SaaS products • Prior experience evaluating AI-generated conversations, chatbots or automated support output • Equal Employment Opportunity Information
Responsibilities
• 1. Evaluate AI Quality Across the Advisor Stack • You are the quality owner for advisor-facing AI inside ORA. You review AI output across the full call lifecycle and surface what is working, what is broken and where the bar needs to be raised. • Review pre-call briefs for accuracy, completeness and signal quality — customer context, adoption signals, retention flags • Evaluate in-call guidance — next-best-action prompts, suggestions and recommendation flows — for relevance and accuracy in live conversation • Score post-call outputs: quality grading, AI-generated summaries, CRM updates, sentiment signals • Audit transcription-driven follow-ups — customer updates, recap notes, follow-up tasks — for accuracy, tone and customer impact • Classify issues by type and severity — hallucination, missing context, wrong action, tone mismatch, KB gap — using standardized rubrics • Participate in calibration sessions to keep scoring consistent across reviewers • 2. Iterate Inside ORA — Light Creation Work • You close the loop. When you spot an issue you can fix without engineering involvement, you fix it inside ORA. • Refine and iterate on prompts within existing skills and sub-agents • Update knowledge base entries to close information gaps surfaced in reviews • Tweak existing skills and orchestrations to improve output quality • Test changes, validate improvements and roll out updates inside ORA • Document what you changed and why — every iteration is a data point • 3. Surface Trends & Drive Continuous Improvement • You connect individual evaluations to systemic insights. Leadership and the Lead Analyst rely on you to know where the next investment should land. • Tag conversations by product area, severity and issue type with consistent classification • Maintain a clear, trusted log of issue types, trends and conversation behaviors • Surface recurring patterns and systemic gaps to leadership and the Lead Analyst • Recommend prioritized improvement opportunities — what to fix, what to build, what to redesign • Build reporting that gives Success leadership visibility into AI quality and the improvement backlog • 4. Partner with RevOps Engineering & the Lead Analyst • When a fix requires ORA platform changes or end-to-end design, you stay close. • Bring structured findings, repro examples and quality criteria when a platform change is needed • Partner with the Lead Analyst on end-to-end builds — bring evaluation signal, test the output, validate the bar • Co-define quality criteria for new sub-agents, skills and orchestrations before they ship • Stay close through the build to ensure improvements land as intended • 5. Stay Sharp: Learn, Experiment, Share • The AI space moves weekly. We hire people who keep up on their own and bring what they learn back. • Stay current on LLM behavior, prompt patterns and AI evaluation techniques • Run small experiments with new prompts, tools or evaluation approaches inside ORA
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT