stninc - Platform Reliability (Horizon AI)

Remote1w ago

Remote WW Platform Engineer Kubernetes Python

Requirements

• Production experience making distributed systems reliable: caching, queuing, circuit breaking, backpressure, timeouts, idempotency • Comfort in Python and containerized environments, with real exposure to Kubernetes or equivalent orchestration • Experience with event streaming and with the failure modes each option brings - Observability depth: metrics, distributed tracing, and the judgment to alert on symptoms rather than causes • A record of shipping reliability improvements with measurable before and after • Nice to have: durable workflow orchestration, blue/green and canary deployment at scale, performance profiling of Python services

Responsibilities

• Build the caching layer that stops every agent tool call from hitting external systems directly • Implement event streaming so services stop depending on synchronous HTTP chains • Harden agent dispatch: health checks, circuit breakers, retry semantics, and observability that distinguishes failure from degradation • Build observability of our own platform: service-level metrics, tracing across services, and alerting we act on • Contribute to the orchestration migration and to infrastructure-as-code • Improve deployment practice: canary, traffic shifting, and real rollback