Hyperbolic Labs - Forward Deployed Infrastructure Engineer

• Cluster stand-up and handoff. Build, validate, and benchmark new customer clusters, then hand them over with documentation the customer's own engineers can work from. • Deep escalations. Multinode and NCCL performance debugging, GPU and hardware faults (XID and ECC errors, lspci, dmesg), driver and fabric issues, container and scheduler problems. • Provider escalation and coordination. Maintenance windows, RMAs, hung nodes, and disputed fault attribution. You bring the evidence that makes the provider act, and you keep the customer informed while it happens. • Embedded ownership of named accounts. You are the engineer your customers know by name. You learn their workload, not just their infrastructure, and you tell them what to change before they hit the wall. • Proactive monitoring. Own the monitoring and alerting we put in front of customer clusters (Grafana, Prometheus) so we find faults before the customer reports them. • Tooling and pushing work down. Automate the repeat work and turn your own escalations into runbooks the L1 tier can run. Anything you fix three times should stop reaching you. • Deep Linux experience and total comfort in the CLI, including in someone else's broken environment. • Hands-on multinode GPU experience: NCCL, InfiniBand or RoCE, collective performance debugging, topology and placement. • Hardware fault triage on GPU nodes: XID and ECC errors, dmesg, lspci, nvidia-smi, thermal and power faults. • Experience provisioning and operating GPU clusters with Kubernetes, Slurm, or both. • Genuinely comfortable customer-facing, including delivering bad news and saying “this is ours” or “this is the provider’s” with confidence. • Sound judgment on when to keep digging and when to escalate. • Experience with parallel filesystems (Weka, Lustre, GPFS) and high-performance storage. • Grafana, Prometheus, or similar observability stacks in production. • Prior forward deployed engineer, solutions architect, or technical account manager experience. • Infrastructure as code (Terraform, Ansible) and CI for cluster provisioning. • Exposure to model training or inference workloads from the practitioner side.