egen - ML Platform Engineer, Distributed Training
Requirements
• 5+ years of engineering experience, with recent hands-on work on GPU training infrastructure and large-scale training. • Strong Python software engineering: production-quality code, tests and code review. Coding is central to this role, not an afterthought. • GKE and GPU scheduling: Kubernetes on Google Cloud with GPU node pools, job queueing (Kueue or similar), and capacity and quota management. • Distributed PyTorch and Ray Train: has personally run and debugged multi-node training jobs. • RDMA and GPU networking: GPUDirect/RDMA, NCCL tuning, multi-node GB200 (or H200) topology. • Benchmarking and evals: has designed performance benchmarks and model evaluations, and used them to make decisions. • Deep enough understanding to explain how and why a system works, without leaning on AI coding tools to carry the reasoning. • A track record of visible ownership: present, communicative, and accountable on a demanding team. • ## Personal Attributes: • Healthcare or life-sciences data (genomics, clinical, biomedical), and working under HIPAA. • Observability and telemetry for training platforms: metrics, tracing, alerting, end-to-end pipeline testing.
Responsibilities
• Training platform • Work with the client's infrastructure team on the GKE training platform (GPU node pools, job queueing and scheduling, quota and capacity) from the training workload's side. • Diagnose and fix platform failures at the root (for example a controller that keeps running out of memory) rather than restarting them. • Tune multi-node GPU networking (RDMA/GPUDirect, NCCL) for training throughput. • Distributed training • Run, scale and debug multi-node distributed training with PyTorch (FSDP, DeepSpeed, Megatron or NeMo-style) and Ray Train. • Profile jobs and remove bottlenecks across compute, memory, input pipelines and interconnect. • Benchmarking and evals • Design and run performance benchmarks for the cluster and training stack: throughput, scaling efficiency, cost per run. • Build and maintain model evaluation harnesses so changes to the platform or training recipes are measured, not assumed. • Instrument training and the platform with telemetry and monitoring, and keep end-to-end pipeline tests running against what users actually run. • Ownership on the client team • Treat the platform as your own: proactively find issues, plug gaps and propose solutions before anyone asks, keep the team informed of status and risks, and look after the scientists who use it. • Explain and defend your design decisions to client engineers and scientists in detail. • Stay effective while the client's priorities and team structure change. • ## Your Technical Toolkit: • Core Languages: Mastery of Python and shell scripting • AI/LLM Ecosystem: Extensive experience with Google Gemini, GPT-4, or LLaMA; deep knowledge of Prompt Engineering and Fine-tuning • Data & Search: Expertise in Vector Databases (Vertex AI Vector Search, pgvector, etc.) and implementing Semantic Search • Infrastructure: Hands-on experience with Google Cloud (Vertex AI) and building scalable software architectures • Frameworks: Proficiency in LangChain, LlamaIndex, or similar orchestration layers • Mindset: A strong software engineering foundation—you write clean, maintainable code and understand the full SDLC
Benefits
• This role is eligible for our competitive salary and comprehensive benefits package to support your well-being: • Comprehensive Health Insurance • Paid Leave (Vacation/PTO) • Paid Holidays • Parental Leave • Bereavement Leave • 401 (k) Employer Match • Employee Referral Bonuses • Check out our complete list of benefits here - >https://egen.ai/people/#benefits • Important: All roles are subject to standard hiring verification practices, which may include background checks, employment verification, and other relevant checks. • EEO and Accommodations:
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT