primeintellect - Member of Technical Staff - Inference
Requirements
• Building ML Systems at Scale: 3+ years building and running large‑scale ML/LLM services with clear latency/availability SLOs. • Inference Backends: Hands‑on with at least one of vLLM, SGLang, TensorRT‑LLM. • Distributed Serving Infra: Familiarity with distributed and disaggregated serving infrastructure such as NVIDIA Dynamo. • Inference Internals: Deep understanding of prefill vs. decode, KV‑cache behavior, batching, sampling, speculative decoding, parallelism strategies. • Full‑Stack Debugging: Comfortable debugging CUDA/NCCL, drivers/kernels, containers, service mesh/networking, and storage, owning incidents end‑to‑end. • Python: Systems tooling and backend services. • PyTorch: LLM Inference engine development and integration, deployment readiness. • Cloud & Automation: AWS/GCP service experience, cloud deployment patterns. • Kubernetes: Running infrastructure at scale with containers on Kubernetes. • GPU & Networking: Architecture, CUDA runtime, NCCL, InfiniBand; GPU‑aware bin‑packing and scheduling across heterogeneous fleets. • Kernel‑Level Optimization: Familiarity with CUDA/Triton kernel development; Nsight Systems/Compute profiling. • Systems Performance Languages: Rust, C++. • Data & Observability: Kafka/PubSub, Redis, gRPC/Protobuf; Prometheus/Grafana, OpenTelemetry; reliability patterns. • Infra & Config Automation: Terraform/Ansible, infrastructure-as-code, reproducible environments • Open Source: Contributions to serving, inference, or RL infrastructure projects.
Responsibilities
• Multi‑tenant LLM Serving: Build a multi-tenant LLM serving platform that operates across our cloud GPU fleets. • GPU‑Aware Scheduling: Design placement and scheduling algorithms for heterogeneous accelerators. • Resilience & Failover: Implement multi‑region/zone failover and traffic shifting for resilience and cost control. • Autoscaling & Routing: Build autoscaling, routing, and load balancing to meet throughput/latency SLOs. • Model Distribution: Optimize model distribution and cold-start times across clusters. • Inference Optimization & Performance • Framework Development: Integrate and contribute to LLM inference frameworks such as vLLM, SGLang, TensorRT‑LLM. • Parallelism and Configuration Tuning: Optimize configurations for tensor/pipeline/expert parallelism, prefix caching, memory management and other axes for maximum performance. • End‑to‑End Performance: Profile kernels, memory bandwidth and transport; apply techniques such as quantization and speculative decoding. • Perf Suites: Develop reproducible performance suites (latency, throughput, context length, batch size, precision). • RL Integration: Embed and optimize distributed inference within our RL stack. • Platform & Tooling • CI/CD: Establish CI/CD with artifact promotion, performance gates, and reproducible builds. • Observability: Build metrics, logs, tracing; structured incident response and SLO management. • Docs & Collaboration: Document architectures, playbooks, and API contracts; mentor and collaborate cross‑functionally.
Benefits
• Flexible work arrangement (remote or San Francisco office) • Full visa sponsorship and relocation support • Professional development budget • Regular team off-sites and conference attendance • Opportunity to shape decentralized AI and RL at Prime Intellect • GROWTH OPPORTUNITY • You'll join a team of experienced engineers and researchers working on cutting-edge problems in AI infrastructure. We believe in open development and encourage team members to contribute to the broader AI community through research and open-source contributions. • We value potential over perfection. If you're passionate about democratizing AI development, we want to talk to you. • Ready to help shape the future of AI? Apply now and join us in our mission to make powerful AI models accessible to everyone.
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT