Staff Software Engineer, Visibility
Upload My Resume
Drop here or click to browse · PDF, DOCX, TXT
Requirements
- 5 or more years of experience as an "Arranger" and/or "Builder/Enhancer" of highly scalable distributed systems. see HERE for more info re: "Arranger" and/or "Builder/Enhancer".
- Strong computer science fundamentals in distributed systems, including concurrency, consistency models, and failure modes
- Experience writing concurrent code in production with languages like Go or Java or other applicable languages with skill level as "high end of Intermediate" and/or "Advanced" or "Expert" levels. see HERE for more info re: "high end of Intermediate" and/or "Advanced" or "Expert levels"
- Hands-on experience designing, operating, and tuning ClickHouse and/or Elasticsearch, ideally in self-hosted environments (managed services are a strong plus).
- Experience building and running services on AWS. Bonus: Azure and/or GCP experience.
- Demonstrated ability to lead large, multi-quarter technical initiatives, especially those involving core data infrastructure and live data migrations.
- Nice to Have • Prior contributions to Temporal, Cadence, or other workflow engines. • Deep expertise in storage internals (e.g., columnar stores, LSM trees, inverted indexes, transactional logs). • Experience operating multi-region services with ≥99.99% uptime. • Strong background in operating and evolving Open Source systems. • Experience building Kubernetes controllers and/or CRDs.
Responsibilities
- Re-architect Temporal Visibility at scale • Lead the design and implementation of a new persistence layer for Temporal Visibility, informed by its real-world access patterns (high-volume writes, time-based queries, filtering, sorting, and pagination across long-running workflows). • Evaluate and select the most appropriate storage technologies (e.g., ClickHouse, Elasticsearch, or complementary systems), clearly articulating tradeoffs around indexing models, consistency, cost, latency, and operational complexity. • Design schemas, APIs, and query models that make Visibility both powerful and intuitive for customers.
- Deliver safe, large-scale data migrations • Plan and execute online migrations of live Visibility data from existing persistence stores to new backends, at scale and without customer downtime. • Design dual-write, backfill, validation, and cutover strategies that prioritize correctness, observability, and rollback safety. • Build tooling and automation to validate data integrity and performance throughout the migration lifecycle.
- Own performance, reliability, and operability • Define and own SLOs for Visibility storage and query paths. • Profile hot paths, design benchmarks, and lead systematic performance tuning efforts. • Build operational playbooks, dashboards, and alerting that make the system understandable and debuggable for on-call engineers. • Lead incident reviews and reliability improvements related to persistence and indexing systems.
- Technical leadership and collaboration • Break down large, ambiguous roadmap initiatives into concrete, executable phases. • Author and steward design docs and RFCs through review with peers and stakeholders. • Mentor and unblock other engineers working in the persistence and storage domain. • Partner closely with Server, Cloud, and Developer Experience teams to land features end-to-end.
Benefits
- The estimated pay range for this role is $180,000 - 260,000.
- Additionally, this role is eligible to participate in Temporal's equity plan.
- Unlimited PTO, 12 Holidays + 2 Floating Holidays
- 100% Premiums Coverage for Medical, Dental, and Vision
- AD&D, LT & ST Disability, and Life Insurance (Standard & Supplemental Available)
- Empower 401K Plan
- Additional Perks for Learning & Development, Lifestyle Spending, In-Home Office Setup, Professional Memberships, WFH Meals, Internet Stipend and more!
- Paid Time Off (PTO) and Benefits outside the United States vary by country, and are issued in partnership with Remote.com. Additionally, Temporal offers perks to all international employees for learning & career development, a lifestyle spending account, in-home office setup (in addition to company-issued hardware), professional memberships, work-from-home meals, and access to the Calm app for mental wellness.
- Travel
- Temporal is a globally distributed, collaborative team that values opportunities for in-person connection. Occasional travel may be required for company events, team offsites, and other meaningful moments that bring us together.
- $3,600 / Year Work from Home Meals
- $1,800 / Year Professional Enrichment (Career Development & Professional Memberships)
- $1,200 / Year Lifestyle Spending Account
- $1,000 / Year In-Home Office Setup (In addition to Temporal issued equipment - laptop, monitor, keyboard, mouse, trackpad, and extension power cable at no cost to you)
- $74 / Month Reimbursement for Internet
- Calm App Subscription for Mental Health & Wellness
Similar Jobs
12h ago