far.ai - Tech Lead Manager, GPU Cluster Infrastructure
Requirements
• You've led engineers as a manager, tech lead, or project lead, setting technical direction, scoping work, and giving feedback. You want to manage people directly; prior direct reports aren't required. • You have 5+ years in systems or infrastructure engineering on production Linux, running GPU, HPC, or large-scale batch platforms, and you've owned systems from design through operation. You have depth in at least one of scheduling, storage, networking, security, or GPU systems, and enough breadth to review designs in the rest. • You've run production Kubernetes for GPU workloads with a batch layer on top (Slurm, Kueue, Volcano, or similar), including quotas, priority and preemption, and node health. • You've owned infrastructure as code and observability for a production fleet, provisioning with Terraform or Ansible, deploying with Helm and ArgoCD, and monitoring with Prometheus, or their equivalents. • You're a strong programmer in at least one language that infrastructure is commonly written in, such as Python, Go, Rust, or C++, and your automation and services are maintained as shared code. • You write clearly for engineers, researchers, and providers, whether it's a roadmap, a design doc, an incident summary, or an escalation. • If you meet most of this and not all of it, we encourage you to apply anyway.
Responsibilities
• Set the platform's technical direction and own its roadmap. You decide which systems we run, how we schedule and store across providers, and what we measure, from utilization and queue wait to failure rates. • Stay in the work. You own architecture and the scheduling and storage design, and you debug the failures that cross layers, such as node health, GPU and fabric faults, and multi-node job hangs. • Hire and grow a small team of senior engineers. You set priorities and ownership, scope projects, and give regular feedback and coaching. • Set security direction for a shared cluster where AI agents run experiments, covering identity and access, workload isolation, and sandboxing. • Decide how we operate, from the on-call rotation and incident response to postmortems, fault tolerance, and observability, and take part in it alongside the team. • Be the escalation point for research teams and for providers, and turn recurring problems into platform fixes.
Benefits
• Health Insurance - 94% of Insurance premium paid by Organization commencing within 1 month after your start date • Retirement - 401(k) plan with up to 2% match • PTO - 25 days Paid Time Off per year, accrued weekly and up to 10 days of paid sick leave per year • Paid Leave - Paid Bereavement, Family, Medical and Pregnancy Disability Leave • WFH Stipend & Equipment - Work computer and stipend provided for eligible employees • Catered Meals (Berkeley Office Only) - Catered lunches and dinners on workdays at our office • (These benefits are currently only offered to full-time US employees, we use an EOR for our international employees who have their own set of benefits for Health, Paid Leave, etc.) • If based in the USA or Singapore, you will be an employee of FAR.AI http://FAR.AI (501(c)(3) research non-profit / non-profit CLG). Outside the USA or Singapore, you will be employed via an EOR organisation on behalf of FAR.AI http://FAR.AI or as a contractor. • Location: Both remote and in-person (Berkeley, CA or Singapore) are possible. We sponsor visas for in-person employees, and can hire remotely in most countries. For this role we prefer candidates whose working hours overlap with Berkeley. • Hours: Full-time (40 hours/week). • On-call: We don't run a formal on-call rotation yet. The team is spread across time zones and covers incidents during working hours. As the experiments we run get larger we expect to introduce one, and this role would take part in it. • Application process: A one hour programming assessment, a short screening call, a one hour code review interview, a 90 minute systems design and incident response interview, a one hour management and leadership interview, and a paid work trial lasting up to 1 week. If you are not available for a work trial we may be able to find alternative ways of testing your fit. • If you have any questions about the role, feel free to contact us at [email protected]. Otherwise, if you don't have questions, the best way to ensure a proper review of your skills and qualifications is by applying directly via the application form. Please don't email us to share your resume (it won't have any impact on our decision). Thank you!
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT