Grafana Labs - Senior Software Engineer

• The SRE team is embedded within the Mimir and Loki squads and focuses on ensuring that Grafana Cloud’s database products deliver exceptional reliability for our highest-SLA customers. We seek a senior engineer operating at the intersection of customer needs, production systems, and product engineering. In this role, you will: • Partner closely with product engineering squads (embedded model) • Own production reliability for high-SLA and complex customer environments • Design and implement automation to scale our reliability practices • Ensuring our customers meet our SLO targets • Define and evolve per-tenant SLOs and reliability models • Proactively reduce SLO burn to prevent repeat incidents • Serving as a primary escalation point and on-call for relevant incidents • Lead customer-impacting incident response and post-incident reviews • Contribute to design docs and code reviews • Influence feature design to ensure production scalability and operability • Build automation to eliminate toil where needed • Improve alert quality and reduce noisy escalations • Of course, there is an on-call component to this role and one that we take seriously. As a company, we hire globally (remote-first) to ensure our on-call remains healthy and aligned to approximately 12 daylight hours per day. You will work closely with counterparts in other regions to provide balanced coverage and shared ownership. • We invest heavily in developer productivity. You can use modern AI coding assistants as part of your daily workflow (your choice of tools, within security guidelines), backed by a company-funded usage budget so you can iterate quickly without unnecessary friction. We encourage pragmatic AI-assisted development: faster prototyping, test generation, refactors, documentation, and incident follow-ups—always paired with strong code review and quality standards. You’ll also have access to frontier models (e.g., GPT-Codex 5/3, Claude Opus 4.6, Gemini 3 Pro). • What Makes You a Great Fit: • Regular 1:1s to with your manager and colleagues • Reviewing and creating SLOs, proactively investigating ways in which we can further reduce budget burn for those SLOs, which can be self-directed or as the result of learnings from incidents, and may include improvements to monitoring, automation, increasing self-healing, auto-scaling, etc. • Improve observability of customers within their environments • Designing and implementing solutions to ensure reliability and scalability of our environments can meet rapidly increasing demands • Develop fault-tolerant design patterns ensuring that we are considering reliability at all stages of the service lifecycle. • Collaborating with our Engineering Leaders to help define and influence product strategy, roadmaps and technical designs • Participate in PR review and collaborating with other engineers on their Design Docs • Teach others about Site Reliability Engineering and communicate best practices to be applied early in development of new features and functionality • Participate in Incident Response when applicable, including investigation through to resolution, PIR, and communication with customers via Bridge calls where necessary