runpod - Senior ML Systems Engineer, Inference
Requirements
• 5+ years of professional system engineering experience. • Deep, hands-on experience with vLLM, SGLang (or a comparable serving engine) in production or at serious benchmark scale.a • Strong software engineering skills in Python. You're comfortable working in large, performance-critical codebases. • A solid understanding of what drives LLM inference performance: batching, memory, parallelism, and the trade-offs between latency and throughput. • Experience with modern inference optimization techniques such as quantization, speculative decoding, or distributed serving. • Rigor in benchmarking and performance analysis, plus comfort with GPU profiling tools. • The ability to explain your results clearly in writing and turn them into decisions. • Experience writing or tuning GPU kernels in CUDA or Triton. • Contributions to inference or ML systems projects. • Experience with multi-node GPU systems and high-speed networking. • Experience at a company where inference cost and latency were core business metrics. • What You’ll Receive: • The competitive base pay for this position ranges from ($150,000 - $220,000). This salary range may be inclusive of several career levels at Runpod and will be narrowed during the interview process based on a number of factors, including the candidate’s experience, qualifications, and location • Meaningful equity in a fast-growing company- everyone on the team receives stock options — your impact drives our growth, and you share in the upside. • Generous medical, dental & vision plans • Flexible PTO- take the time you need to recharge • Most roles are remote work first with an inclusive, collaborative teams utilizing slack as the main form of internal communication • Join a passionate team on the cutting edge of AI infrastructure — where culture, learning, and ownership are at the heart of how we scale. • $1,200 Home Office & Equipment Stipend- We set you up for success from day one with gear and support to create your ideal workspace
Responsibilities
• Define how we measure inference performance, including throughput, time to first token, inter-token latency, and cost per token, and build the tooling that makes those measurements rigorous and repeatable. • Profile and diagnose performance problems across the serving stack, from scheduling and memory management down to kernels and interconnect. • Improve serving efficiency for large, state-of-the-art models on single-node and multi-node GPU deployments. • Turn what you learn into production-ready runtimes, configurations, and defaults that customers benefit from automatically. • Work closely with product and infrastructure teams to shape how inference is offered on Runpod. • Keep up with the fast-moving inference ecosystem, including the open-source community, and decide what's worth adopting, what's worth building, and what's worth contributing back. • Trace bottlenecks in the serving engine/runtime and implement fixes when configuration tuning is not enough.
Benefits
• Decline to self-identify • Hispanic or Latino - A person of Cuban, Mexican, Puerto Rican, South or Central American, or other Spanish culture or origin regardless of race. • Hispanic or Latino • White (Not Hispanic or Latino) - A person having origins in any of the original peoples of Europe, the Middle East, or North Africa. • White • Black or African American (Not Hispanic or Latino) - A person having origins in any of the Black racial groups of Africa. • Black or African American • Native Hawaiian or Other Pacific Islander (Not Hispanic or Latino) - A person having origins in any of the peoples of Hawaii, Guam, Samoa, or other Pacific Islands. • Native Hawaiian or Other Pacific Islander • Asian (Not Hispanic or Latino) - A person having origins in any of the original peoples of the Far East, Southeast Asia, or the Indian Subcontinent, including, for example, Cambodia, China, India, Japan, Korea, Malaysia, Pakistan, the Philippine Islands, Thailand, and Vietnam. • Asian • American Indian or Alaska Native (Not Hispanic or Latino) - A person having origins in any of the original peoples of North and South America (including Central America), and who maintain tribal affiliation or community attachment. • American Indian or Alaska Native • Two or More Races (Not Hispanic or Latino) - All persons who identify with more than one of the above five races. • Two or More Races • Hispanic or Latino • White (Not Hispanic or Latino) • Black or African American (Not Hispanic or Latino) • Native Hawaiian or Other Pacific Islander (Not Hispanic or Latino) • Asian (Not Hispanic or Latino) • American Indian or Alaska Native (Not Hispanic or Latino) • Two or More Races (Not Hispanic or Latino) • I identify as one or more of the classifications of protected veteran listed above • I am not a protected veteran
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT