Cobalt - Staff AI Engineer
Requirements
• Hands-on experience building and operating LLM-based agents in production, with a clear understanding of their scale, constraints and business outcomes. • A track record of improving agent reliability: building evaluations, investigating non-deterministic failures and improving success rates against measurable goals. • Strong knowledge of agent architectures, including tool use, orchestration, planning, context and memory management, and the judgment to know when simpler workflows are sufficient. • Experience running LLM workloads through managed cloud model platforms or direct provider APIs, including rate limits, failure handling and cost tracking. • Strong backend and distributed systems fundamentals, including APIs, asynchronous workflows, concurrency and fault tolerance. • Experience with production observability, incident investigation and safe rollout of system changes. • An understanding of security risks in agent systems, including prompt injection, unsafe tool execution and exposure of sensitive data. • Evidence of impact beyond your own code: shared platforms other teams adopted, standards that improved outcomes or engineers you helped grow. • Tools and Practices You’ve Likely Used • We care more about your depth of experience and engineering judgment than experience with any specific tool. • Agent orchestration: Framework-based or custom orchestration, tool calling and structured outputs. • Agent orchestration: • Tool integration: Model Context Protocol (MCP), APIs and permission-controlled tool execution. • Tool integration: • Evaluations: Automated evaluation harnesses, domain-specific test datasets and regression testing in CI. • Evaluations: • Observability: Distributed tracing, structured logs and metrics for multi-step agent workflows. • Observability: • Model access: Managed cloud model platforms, direct provider APIs and multi-provider gateways. • Model access: • Experience with GCP, Go or Kubernetes. • Experience building sandboxed environments for agents that execute untrusted code or interact with external systems. • Offensive security, application security or security tooling experience. • Experience introducing AI coding tools across engineering teams and measuring their impact. • Public writing, talks or open-source contributions related to agent engineering.
Responsibilities
• Set technical direction for agent systems across teams, including orchestration, tool use, context and memory management, human oversight and failure recovery. • Design, build and operate shared foundations: evaluation frameworks, agent observability and tracing, execution controls, and model access with routing and failover where needed. • Establish evaluation practices that measure task completion, accuracy, false positives, security and cost per successful task. Work with domain experts to build representative test cases and turn production failures into regression tests. • Make agent workflows resilient to tool failures, timeouts, rate limits and partial execution, with clear limits on retries, runtime and spend. • Work with security and platform teams to build in scoped permissions, customer data isolation, audit trails and safe execution of tools and code. • Establish practical engineering standards: evals in CI, prompt and model versioning, regression testing for non-deterministic systems, cost and latency budgets, and staged rollouts with rollback paths. • Lead model selection and cost optimization. Benchmark models against real workloads and make informed trade-offs between quality, latency and cost. • Partner with teams on production readiness, incident investigation and improvements to agent reliability. • Help teams use AI coding agents effectively across planning, implementation, review and testing. Measure the impact on delivery speed and software quality. • Mentor engineers and technical leads through design reviews, pairing, internal talks and written guidance. Help teams adopt shared capabilities and improve them based on feedback. • Work with product, research and security leaders to turn business priorities into a technical roadmap with clear outcomes.
Benefits
• Grow in a passionate, rapidly expanding industry operating at the forefront of the Pentesting industry • Work directly with experienced senior leaders with ongoing mentorship opportunities • Earn competitive compensation and an attractive equity plan • Save for the future with a 401(k) program (US) or pension (EU) • Benefit from medical, dental, vision and life insurance (US) or statutory healthcare (EU) • Leverage stipends for: • Work-from-home equipment & wifi • Learning & development • Make the most of our flexible, generous paid time off and paid parental leave • Cobalt is committed to fair and equitable compensation practices. The salary range for this role is ($200,000 - $240,000) + equity + benefits. A candidate’s salary is determined by various factors including, but not limited to, relevant work experience, skills, and certifications. The salary range may differ in other states and may be impacted by proximity to major metropolitan cities.
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT