Breakmark - AI Compute Engineer
Requirements
• Strong hands-on experience with NVIDIA GPU infrastructure, including HGX or comparable large GPU server platforms. • Demonstrated experience designing, executing, or leading burn-in and qualification across multi-node GPU clusters on H100, H200, B200, B300, or a comparable platform (direct B300 experience not required). • Experience analyzing fleet-level qualification results, performance distributions, and hardware/software failures. • Strong Linux systems administration and troubleshooting. • Experience with NVIDIA drivers, CUDA, NCCL, DCGM, and GPU platform tooling. • Experience diagnosing GPU, NVLink, NVSwitch, PCIe, thermal, power, or firmware problems. • Experience validating intra-node and multi-node GPU communication, plus a working understanding of RDMA, GPUDirect RDMA, and high-performance AI fabrics. • Experience deploying, provisioning, or operationalizing GPU clusters. • Experience defining firmware, BIOS, OS, and software baselines, as well as health, quarantine, readiness, and return-to-service criteria. • Experience interpreting distributed training or inference workload behavior. • Ability to troubleshoot performance across GPU, host, network, storage, and software boundaries. • Ability to define requirements for automation teams and interpret the outputs of automated validation systems. • Strong root-cause analysis and problem-solving skills. • Clear communication of technical findings, risk, and remediation requirements. • Comfortable in fast-moving deployment, commissioning, and production environments. • NVIDIA B200 or B300 infrastructure. • ConnectX adapters, NVIDIA Quantum, InfiniBand, UFM, RoCE, or comparable networking. • Large-scale NCCL and distributed workload qualification. • PyTorch, NeMo, Megatron, vLLM, or TensorRT-LLM. • GPU schedulers such as Slurm, Kubernetes, Run:ai, Base Command, or equivalent. • Validating shared storage and distributed checkpointing on high-performance platforms such as WEKA, VAST, or equivalent. • Infrastructure automation, observability, telemetry, or fleet-management platforms. • Formal customer acceptance testing. • Operating GPU infrastructure across multiple data centers. • Python, Bash, or equivalent scripting for troubleshooting infrastructure workflows. • What We ValueEngineers who have qualified and operated real GPU clusters and who understand what a test reveals, not just how to run it. We value evidence-based judgment, strong ownership of platform integrity, cross-domain leadership without overreach, disciplined and reproducible operations, clear communication of risk and uncertainty, and curiosity across GPU generations. • What We Value
Responsibilities
• Define the architecture: Set lifecycle standards for NVIDIA HGX platforms and validated baselines for firmware, BIOS, BMC, OS, drivers, CUDA, and NCCL. Own compatibility requirements and the model of GPU/CPU/memory/PCIe/NVLink/NVSwitch/NIC/storage relationships. Decide what testing each platform change requires before production. • Define the architecture: • Lead qualification and burn-in: Design the strategy for new clusters, new GPU platforms, repaired nodes, and major changes. That includes load profiles, duration, coverage, and pass criteria. Analyze fleet-wide distributions and outliers to separate isolated failures from systemic issues in hardware, firmware, software, topology, power, thermals, or test configuration. • Lead qualification and burn-in: • Diagnose health and performance: Interpret NVIDIA diagnostics and telemetry, covering GPU, NVLink, NVSwitch, PCIe, memory, network, thermal, and power signals. Lead root-cause analysis for intermittent, multi-node, and fleet-wide issues, including collective and point-to-point communication problems. • Diagnose health and performance: • Validate with real workloads: Define representative training and inference workloads, including dense, MoE, and different parallelism strategies. Set reproducible baselines for throughput, scaling, latency, and numerical behavior, and pinpoint whether degradation comes from compute, communication, storage, scheduling, or configuration. • Validate with real workloads: • Integrate across domains: Work with Network (topology, GPU-to-NIC locality, RDMA), Storage (access, checkpointing, recovery), Platform/DevOps (scheduler, containers, provisioning), and Facilities (power, thermal correlation). Route issues to the right specialists with solid evidence. • Integrate across domains: • Own operational readiness and acceptance: Define health, readiness, quarantine, and return-to-service gates. Guide automation teams in building reproducible workflows and interpret their results. Translate customer requirements into validation strategies, rule platforms ready, degraded, conditional, or unsuitable, define remediation and re-test scope, and communicate risks to engineering, vendors, customers, and leadership. • Own operational readiness and acceptance:
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT