Telnyx - Inference Infrastructure Architect (Remote)

• Operate and expand the fleet efficiently. Maximize useful inference throughput per GPU-dollar while meeting latency and reliability SLOs, and continuously reduce cost per token. • Operate and expand the fleet efficiently. • Make the product run on it. Serverless inference for the open-weight catalog, and dedicated, tuned model deployments for enterprises. • Make the product run on it. • The stack is open source, bare metal to OpenAI-compatible endpoint. You work upstream in it. • What you'll build • What you'll build • Serverless serving pools. vLLM / SGLang engines with continuous batching, prefix caching, low-precision serving (FP8, FP4, INT4) and MoE expert parallelism, serving the head of the open-weight catalog at multi-tenant scale. • Serverless serving pools. • The fleet layer. llm-d / NVIDIA Dynamo on the Kubernetes Gateway API: KV-cache-aware routing, prefill/decode disaggregation, KV tiering (Mooncake), and multi-LoRA serving so hundreds of customer adapters share one base pool. • The fleet layer. • The platform under it. Kubernetes on bare metal: GPU Operator, topology-aware scheduling, LeaderWorkerSet, Kueue; Kata Containers for isolated dedicated tenants; bare-metal lifecycle with OpenStack Ironic. • The platform under it. • Weight logistics and elasticity. P2P model distribution (Dragonfly, safetensors streaming), warm pools, and autoscaling driven by inference metrics — queue depth, KV occupancy, TTFT — never CPU. • Weight logistics and elasticity. • The dedicated tier. Per-tenant pools, GPU-hour metering, latency SLOs, private networking; and the serving side of the model-tuning loop: adapter versioning, canary rollout, rollback on outcome regression. • The dedicated tier. • Observability and economics. DCGM and engine metrics into Prometheus / OpenTelemetry; capacity planning grounded in roofline math — bandwidth-bound decode, batching curves, utilization versus cost per token. • Observability and economics. • The stack you'll work in • Some of this is committed direction: Kubernetes on bare metal, vLLM / SGLang, an OpenAI-compatible endpoint. Much of the rest is candidates you'll evaluate. You'll select, benchmark and integrate the components that earn their place in production. We value depth in the core serving stack and sound architectural judgment, not prior experience with every project listed. • Engines: vLLM · SGLang • Engines: • Serving techniques: prefill/decode disaggregation · wide expert parallelism · speculative decoding (MTP, EAGLE-3) · low precision (FP8, FP4, INT4) • Serving techniques: • MoE & attention libraries: DeepEP · DeepGEMM · EPLB · FlashMLA • MoE & attention libraries: • Routing & serving: llm-d · NVIDIA Dynamo · Kubernetes Gateway API · Envoy · Ray Serve / KubeRay • Routing & serving: • KV cache & weights: Mooncake · HiCache · 3FS · Dragonfly • KV cache & weights: • Scheduling & GPU sharing: Kubernetes · Volcano · Kueue · LeaderWorkerSet · HAMi • Scheduling & GPU sharing: • Isolation & bare metal: Kata Containers · OpenStack Ironic · NVIDIA GPU Operator • Isolation & bare metal: • Observability: Prometheus · Grafana · DCGM · OpenTelemetry • Observability: • What we look for • What we look for • You have owned production LLM serving under meaningful traffic and latency constraints, and can explain the architecture, the bottlenecks you diagnosed, the changes you made, and the resulting improvements in latency, reliability or cost. Scale on the order of thousands of GPUs or millions of requests a day is a strong signal, not a hard requirement. • Kubernetes on GPU fleets, operated end to end: GPU Operator, device plugins, node pools and topology-aware placement, gang scheduling (LeaderWorkerSet, Kueue), GitOps rollouts, and the ability to debug why a pod landed where it did. • Deep operational command of vLLM or SGLang: deploying, tuning and upgrading it (parallelism with TP / EP, quantization, batch and KV-cache settings, prefix caching, disaggregation), and knowing which knob moves which metric. • Performance engineering at the system level: you read engine and DCGM metrics, reason from roofline, and size deployments with numbers. • Python and Go for automation; real Linux, networking and storage depth. You don't need to write CUDA; you need to know when a problem is one, and route it upstream. • You work with the community in English and Chinese, and write runbooks and design docs people actually read. • Operated inference platforms at the scale of Ant Group, Alibaba Cloud / PAI, Tencent, ByteDance, Baidu, Huawei, DaoCloud, Moonshot, DeepSeek, or comparable teams. • Contributor to, or heavy production user of, any of: vLLM, SGLang, llm-d, NVIDIA Dynamo, Ray / KubeRay, Kata Containers, Volcano, Kueue, HAMi, Dragonfly, Mooncake, Envoy. • Presence in the CNCF / OpenInfra community: talks at KubeCon China, OpenInfra Days, vLLM or SGLang meetups. • Fine-tuning / RL infrastructure: LoRA pipelines, evaluation harnesses, champion/challenger rollout. • Real-time voice latency work: sub-second time-to-first-token budgets on live calls. • Multi-region deployments and data-residency requirements.