PhysicsX - Platform Engineer - Self Hosting LLMs built with GO
Requirements
• Where you need to be strongest • You have operated inference engines such as vLLM or SGLang, or serving platforms such as KServe, under production traffic, latency and reliability constraints. • You can explain the boundary between serving orchestration and inference execution, including how deployment, scaling and runtime configuration affect performance. • You have built or operated model routing across more than one provider, and can explain how you balanced cost, quality, latency, failure and policy. • You can describe the serving architecture you operated, the bottlenecks you found, the changes you made and the measured effect on latency, throughput, reliability or cost. • You understand LLM inference as a systems problem: batching, scheduling, KV-cache behaviour, memory pressure, model parallelism and GPU utilisation. • You have built and operated GPU workloads on Kubernetes, including deployment, scaling, observability and failure recovery. • You can turn fast-moving open-source engines and provider APIs into a supportable product with versioning, compatibility testing, rollback and clear operational boundaries. • You are a strong systems engineer who remains hands-on in architecture, implementation, profiling and production debugging. • Your experience goes materially beyond local model runners or demonstrations; tools such as Ollama are not a substitute for production serving ownership. • Foundations • Strong Python plus a systems language such as Go, Rust or C++. • Linux, containers, Kubernetes, networking, storage and distributed-systems fundamentals. • GPU observability and performance analysis across engine, system and accelerator metrics. • Model formats, quantisation, tokenisation and OpenAI-compatible serving APIs. • Traffic management, streaming, overload protection, health checking and failure recovery. • Integration with existing identity, authorisation, secrets and workload-isolation systems. • Clear technical writing, design review, incident response and cross-team communication. • Contributions to KServe, vLLM, SGLang or adjacent inference, gateway or orchestration projects. • Experience modifying engine internals, profiling kernels or diagnosing CUDA and collective-communication issues. • Experience with mixture-of-experts models, expert parallelism, speculative decoding or disaggregated serving. • Experience routing by data-sensitivity policy, evaluation signals or multi-model review workflows. • Experience with model serving in customer-operated, regulated or fully disconnected environments. • Familiarity with Kata Containers, confidential computing or strong multi-workload isolation. • Experience integrating serving with post-training rollouts and evaluation workloads. • We value diversity and are committed to equal employment opportunity regardless of sex, race, religion, ethnicity, nationality, disability, age, sexual orientation or gender identity. We strongly encourage individuals from groups traditionally underrepresented in tech to apply. To help make a change, we sponsor bright women from disadvantaged backgrounds through their university degrees in science and mathematics. • We collect diversity and inclusion data solely for the purpose of monitoring the effectiveness of our equal opportunities policies and ensuring compliance with employment and equality legislation. This information is confidential, used only in aggregate form, and will not influence the outcome of your application.
Responsibilities
• Make self-hosted serving and model routing work in air-gapped deployments, including sites where the hardware, connectivity and operational setup force constrained solutions. • Own LLM packaging, registration, deployment, health checks, upgrades, rollback and retirement for supported LLMs. • Evaluate and integrate serving platforms such as KServe, including inference services, serving runtimes, model loading and rollout controls. • Define, model, and implement service levels for availability, quality, and performance metrics (e.g. time to first token, inter-token latency, and tail latency). • Measure and improve throughput, GPU utilisation, memory use and cost per token under representative agent workloads. • Tune continuous and chunked batching, KV and prefix caching, quantisation, speculative decoding and tensor, data or expert parallelism where appropriate. • Evaluate advanced techniques such as cache-aware routing and prefill/decode disaggregation when workload evidence justifies their complexity. • Build autoscaling, admission control and capacity management around GPU and serving signals rather than generic CPU metrics. • Make every routing decision observable and explainable, and use evaluation evidence before adding specialist routes. • Work with Research and Post-Training specialists to move candidate and tuned models through evaluation into supported production routes. • Help determine how this capability should grow and, over time, shape the team around it.
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT