horizon3ai - Staff Software Engineer, OTA Services
Requirements
• Distributed Systems & Delivery • 8+ years of professional software engineering experience with deep specialization in distributed systems, remote update systems, or fleet management infrastructure • Deep understanding of remote update system design — delivery protocols, version negotiation, delta delivery, health verification, and rollback semantics • Experience designing rollout strategies for software deployed across heterogeneous, customer-managed environments — you understand the difference between a lab rollout and rolling something out to thousands of production deployments you don't control • Strong proficiency in Go, Python, or Rust for systems-level infrastructure work • Platform & Infrastructure • Experience building software that operates in constrained or air-gapped environments — you know how to design delivery systems that don't assume good connectivity • Deep understanding of Kubernetes — how deployments are managed, updated, and monitored in running clusters • Experience with container image distribution: OCI artifacts, image manifests, layer caching, and registry protocol • Strong understanding of mTLS, certificate lifecycle, and secure channel establishment for update delivery • Technical Leadership • Track record of setting engineering standards that other engineers adopt — protocol designs, reliability requirements, and testing standards for distributed systems • Ability to produce clear architecture documents and drive technical decisions across team boundaries (especially with the Image Build team that produces your input artifacts) • Experience mentoring mid-level engineers through design review and code review • Experience with Kubernetes operators or controller patterns — OTA delivery in Kubernetes environments often benefits from operator-style lifecycle management • Background in cybersecurity or experience building software for regulated or high-security environments • Familiarity with Fleet (or similar open-source fleet management tools) and where custom infrastructure is preferable • Experience with signed artifact delivery and chain-of-custody verification end-to-end • We are a fully remote company, and this job may require up to 5% of travel to be successful.
Responsibilities
• The Expert Engineer on the OTA Services team is the technical anchor for how NodeZero updates itself across thousands of customer deployments — without customer intervention, without downtime, and without introducing new vulnerabilities in the process. NodeZero runs as a container inside customer infrastructure. When a new image is ready, the OTA system has to find every deployed instance, negotiate the update, deliver it safely, validate it, and roll back gracefully if something goes wrong. Getting that right at scale, across diverse customer environments including air-gapped and restricted networks, is a genuinely hard systems problem. • This is a hands-on staff-level role. You will write production code, produce architecture designs, set the technical direction for update delivery and version management, and mentor P4 engineers on the team. The OTA Services team works in close partnership with the Image Build & Release team — your counterpart on the build side produces the artifacts you deliver. This role reports to the Client Platform Engineering Manager. • OTA Architecture: Design and own the over-the-air update system — the delivery protocol, version negotiation, rollout orchestration, health verification, and automatic rollback • Deployment Fleet Management: Own the systems that track the version state of every NodeZero deployment — know what's running where, what's pending, and what's stuck • Rollout Strategy: Design safe rollout strategies for updates — phased rollouts, canary deployments, customer pinning, and emergency hold mechanisms • Constrained Environment Delivery: Design delivery mechanisms that work reliably across diverse customer environments including air-gapped networks, strict egress policies, and minimal-footprint deployments • Reliability & Incident Response: Own OTA system reliability end-to-end — on-call for delivery failures, stuck updates, and version drift incidents • Mentorship: Mentor P4 engineers through design review, code review, and technical pairing; be the person the team turns to when delivery semantics get complicated • Application Note • In any materials you submit, you may redact or remove age-identifying information such as age, date of birth, or dates of school attendance or graduation. You will not be penalized for redacting or removing this information.
Benefits
• At Horizon3, we believe that our people are our greatest asset, and our compensation philosophy reflects this core value. We are committed to fostering an environment where all employees feel valued, respected, and rewarded for their contributions. Our compensation structure is designed to be fair, competitive, and transparent, ensuring that every team member is recognized and compensated equitably across roles, levels, and locations. • In accordance with various State's transparency regulations, we provide the following salary range information for this position: • Base salary range: $229,500 - $260,000 annually. The exact salary will be determined based on the selected candidate's location, qualifications, experience, and relevant skills. • Additional compensation: This role may also be eligible for an equity package (in the form of stock options). If any other compensation benefits apply, they will be discussed during the interview process. • Inclusive Team: We value diversity and promote an inclusive culture where everyone can thrive. • Growth Opportunities: Be part of a dynamic and growing team with numerous career development opportunities. • Innovative Culture: Work in a collaborative environment that encourages creativity and out-of-the-box thinking. • Remote Work: We are a 100% remote company. Enjoy the flexibility to work in the way that supports you and brings out your best. • Competitive Compensation: We offer competitive salary and benefits which includes health, vision & dental care for you and your family, a flexible vacation policy, and generous parental leave. • You Belong Here
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT