Causal Labs - Member of Technical Staff — Training Infrastructure

San Francisco, California, United States3w ago

In Office Staff NA Artificial Intelligence Nonprofit Training Specialist JAX PyTorch

Requirements

• We value a relentless approach to problem-solving, rapid execution, and the ability to quickly learn in unfamiliar domains. • Demonstrated proficiency with distributed training frameworks and techniques (e.g. FSDP, DeepSpeed, Megatron, Pytorch, JAX/XLA) to train large foundation models • Strong grasp of state-of-the-art techniques for optimizing training workloads: parallelism strategies, memory optimization, mixed precision, communication overlap • Ability to profile and debug performance in complex codebases, from framework internals down to kernels and collectives • Deep understanding of deep learning frameworks (e.g. PyTorch, JAX) and their underlying system architectures • Bonus: contributions to open-source ML infrastructure (e.g. PyTorch, Megatron-LM, DeepSpeed, XLA)

Responsibilities

• Design, implement, and optimize distributed training systems that scale across thousands of GPUs • Research and test parallelization strategies and numerical precision trade-offs across model scales, including for architectures that don't map cleanly onto existing LLM training stacks • Analyze, profile, and debug low-level GPU operations to maximize throughput and hardware utilization • Build reusable frameworks for checkpointing, fault tolerance, and reproducibility that stay robust under rapid research iteration • Collaborate with researchers to bring novel model architectures from prototype to full scale • Stay up-to-date on research to bring new ideas to work