Backblaze External Website - Sr. Observability Systems Engineer
Requirements
• 8+ years in a systems engineering, SRE, DevOps, or observability-focused role, including demonstrated technical leadership on a platform or infrastructure team. • Deep hands-on experience with Prometheus, VictoriaMetrics, or equivalent metrics platforms, including PromQL, Alertmanager, and operating these systems at scale. • Experience designing and operating log aggregation pipelines such as Elasticsearch/OpenSearch, Logstash, Filebeat, or similar. • Working knowledge of distributed tracing (Jaeger, OpenTelemetry, or equivalent), including instrumentation strategy, not just operation. • Proficiency in Python and/or Go for automation and tooling. • Experience with Kubernetes, including deploying workloads and understanding pod networking and storage. • Experience with infrastructure-as-code tooling and configuration management. • Strong understanding of Linux internals, with Debian/Ubuntu preferred. • Demonstrated experience mentoring engineers and raising the technical bar on a team. • Excellent written and verbal communication skills. You'll work across engineering, SRE, security, and leadership, and you'll be expected to write design documents that drive decisions. • Experience leading the evaluation or deployment of AIOps tooling, including alert correlation, deduplication, or automated root cause analysis. • Experience building, deploying, or operating AI agents or agentic AI systems. This is a strong differentiator. • Familiarity with LLM APIs (Anthropic, OpenAI) or self-hosted LLM inference in an infrastructure context. • Familiarity with Ceph storage, including OSD management, pool configuration, and CRUSH maps. • Experience with incident management platforms. • Familiarity with BGP, network observability, or working in multi-DC environments. • Experience with secrets management in production environments. • Experience operating in a regulated or compliance-driven environment. • Prior experience in a storage-focused company or hyperscale infrastructure environment. • What You'll Work On • What You'll Work On • A telemetry stack spanning several data centers with exabytes of storage under observation. • Real infrastructure at scale: Kubernetes, Ceph, Vitess, KVM, and a custom storage platform serving hundreds of thousands of customers. • A collaborative team that values engineering excellence, transparency, and operational rigor. • Pay Transparency • Pay Transparency • To provide greater transparency to candidates, we share base pay ranges for all US-based job postings regardless of state. Final offer amounts are determined by multiple factors including candidate location, skills, depth of work experience, and relevant licenses/credentials. • The expected salary range for this role is $178,000 - $200,000 plus 10% target annual bonus and RSUs.
Responsibilities
• Technical Leadership • Set technical direction and standards for observability across the organization, including instrumentation conventions, alerting philosophy, and dashboard design patterns. • Mentor engineers on the Observability team and act as a technical resource for SRE, Storage Engineering, and Network Engineering on observability practice. • Lead design reviews for observability work and represent the team in cross-functional architecture discussions. • Evaluate build-versus-buy decisions for observability and AIOps tooling, including data sovereignty and compliance considerations. • Produce and maintain design documentation, runbooks, and architectural decision records that make the platform operable and extensible by future contributors. • Observability Platform • Own and evolve our metrics pipeline, including VictoriaMetrics, Prometheus, and the Consul-based service discovery layer that keeps scrape targets current as infrastructure scales. • Own and expand our logging stack (OpenSearch, Logstash, and Filebeat) across all Backblaze data centers. • Architect and expand distributed tracing with Jaeger and OpenTelemetry, extending coverage beyond the current application instrumentation to new services. • Define and maintain Grafana dashboard standards for infrastructure, storage, and application performance visibility. • Drive expansion of internal application telemetry in partnership with application teams to improve visibility. • Define capacity and retention strategy for the telemetry stack as the fleet grows. • Infrastructure & Automation • Own infrastructure-as-code for observability stack deployments across multiple Kubernetes clusters. • Write and review automated playbooks for configuration management and automation across Debian-based bare metal servers. • Partner with the SRE team on Consul-based service discovery for VictoriaMetrics scrape target automation. • Lead Ceph storage observability, including OSD health, performance metrics, and integration with alerting. • Collaborate with the Network Engineering team on BGP, routing, and network-level observability.
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT