whitecircle - ML Infrastructure Engineer

• Build robust, flexible, and scalable RL and post-training pipelines, including smoke tuning runs for quality testing and approach ablations • Design data control systems that govern what the model sees, when it sees it, and how training data flows through rollouts, replay, filtering, evaluation, and policy updates • Tune training and inference end-to-end for high throughput across the systems that matter: networking, memory, compute scheduling, data loading, storage, checkpointing, and I/O • Investigate how infrastructure choices affect learning dynamics, eval quality, model behavior, and training stability – staying close to the state of the art in LLMs, RL, and post-training • Build infrastructure for model iteration: experiment runs, artifacts, evals, dashboards, failure inspection, reproducibility, and cost visibility • Work on inference infrastructure where it affects post-training and evaluation loops • Build and improve agentic development environments: coding-agent harnesses, browser/tool integrations, terminal/runtime sandboxes, repo-aware workflows, and multi-agent orchestration • Work closely with the team: plan future steps, discuss tradeoffs, share context early, and stay in touch while building • You’ll fit right in if you: • Have designed, built, or maintained distributed RL/post-training systems at scale and are fluent in their moving parts: rollouts, replay buffers, reward signals, data filtering, policy updates, evaluation loops, and failure analysis • Are familiar with deep learning frameworks such as PyTorch or JAX • Are proficient in Python, including concurrency, asynchronous programming, multiprocessing, and performance optimization • Can debug distributed GPU workloads across CUDA runtime, container runtime, driver versions, NCCL or equivalent communication layers, networking, storage, scheduling, and checkpointing • Have experience with profiling tools across the stack, for example py-spy, PyTorch profiler, Nsight, perf, tracing, metrics, logs, or custom instrumentation • Have experience with inference stacks such as vLLM, SGLang, TensorRT-LLM, Dynamo, or custom serving infrastructure • Can reason from system metrics back to model behavior: when latency, queueing, sampling, data order, rollout throughput, or infrastructure failures affect learning • Have a strong ownership mindset: you can take an ambiguous infrastructure problem, make it concrete, ship a working system, and improve it from real feedback • A public builder footprint: open-source contributions to RL, distributed ML, LLM training, inference, eval, or agent infrastructure – repos, PRs, benchmarks, papers with code, technical posts – and a good technical X/Twitter presence with live building, debugging threads, and useful interaction with strong builders • Experience in a high-bar AI infra, research, or model environment such as xAI/Grok, Qwen, ByteDance AI infra/research, Prime Intellect, or similar teams • Custom training framework support or ownership: distributed training, fine-tuning pipelines, trainers, schedulers, checkpointing, data loaders, model/eval integration, or performance tooling • Serious use of Claude Code, Codex, Kimi Code, Pi Agent, Droid, or similar agentic coding systems as a development surface • Experience with GPU clusters on Kubernetes, Slurm, Ray, custom schedulers, or cloud GPU orchestration • NCCL, UCX, NVSHMEM, RDMA, InfiniBand, RoCE, or EFA • Rust, C++, CUDA, Go, or systems-level performance work