Andromeda Cluster - Customer Reliability Engineer
Requirements
• Real Linux troubleshooting ability from the command line. You can work a problem through logs, processes, networking, and disk without a script to follow • Working knowledge of Kubernetes: pods, nodes, deployments, services, scheduling, and how to investigate when one of those breaks • Can write a script in Python or Bash to automate something repetitive • Strong writing. You can explain a technical problem to a frustrated customer clearly and without condescension • Good judgment under pressure. You know what to check first, when to escalate, and how to keep people informed while you're still working it out • Willing to join a 24/7 on-call rotation • Strong Candidates May Have • Hands-on time with NVIDIA GPUs in production: drivers, CUDA, DCGM, the Kubernetes device plugin • Experience with high-performance networking (InfiniBand, RoCE) or NCCL • Experience with HPC or batch schedulers like Slurm • A previous customer-facing technical role: support engineering, TAM, solutions, professional services • Knowledge of Prometheus, Grafana, Datadog, or similar • IAC: Terraform, Ansible, or Helm • Genuine interest in AI infrastructure and how big training jobs behave
Responsibilities
• Triage and fix customer issues • Own issues start to finish: reproduce, diagnose, fix or escalate, close the loop • Debug at the Linux layer: processes, networking, storage, kernel logs, resource contention, systemd, journald • Dig into Kubernetes problems like pods stuck pending or crash-looping, node conditions, scheduling failures, resource limits • Work GPU failures: driver and device-plugin issues, XID errors, thermal throttling, nodes that need cordoning or draining, jobs failing across multiple nodes • Escalate when you're past your depth, with the evidence already gathered • Handle incidents • Take part in a 24/7 on-call rotation • First response on alerts and customer-reported outages: assess impact, set severity, pull in the right people • Keep customers updated during incidents. Clear status, honest unknowns, no silence • Write up what happened, then turn it into a runbook, an alert, or a fix so it costs less next time • Push providers to resolution • Work out whether a fault is provider-side, ours, or the customer's before it gets handed anywhere • Open tickets with compute providers and chase them down rather than waiting • Track recurring provider failures and flag the patterns to the people making sourcing decisions • Build the tooling • Write Python or Bash to automate the checks you'd otherwise run by hand • Build and improve monitoring: cluster and node health checks, GPU telemetry, dashboards, alerts that fire on real problems • Keep runbooks and customer docs current as you go
Benefits
• High-growth environment: Get in early at a company at the center of the AI infrastructure boom • Competitive compensation: + meaningful equity • Comprehensive benefits: for you and your dependents, including healthcare, dental, and vision coverage, 401(k), and unlimited PTO
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT