Senior Inference SW Engineer

Description

The company is a global edge-to-cloud business. The Private Cloud AI group is hiring a Senior Software Engineer in Engineering (job level TCP04) to build and extend the model runtime in the company's inference platform. Enterprises use the platform to run large language models on hardware they own, including air-gapped and sovereign sites.

The hard part is not deploying models. It is keeping them fast and the GPUs busy on customer hardware that differs in generation and configuration. You will build parts of the runtime, including engine integration, batching, KV cache handling and distributed execution, and you will work on the Kubernetes layer that orchestrates it.

This role is hybrid. You will work from a company office on average 2 days per week.

Responsibilities

• Design, build and take ownership of major parts of the LLM serving stack, such as engine integration, continuous batching, KV cache management and reuse, and quantized execution.
• Work with inference engineering teams to lower time-to-first-token, inter-token latency and P95/P99 tail latency, and to raise throughput per GPU.
• Build and run distributed execution features, including disaggregated prefill and decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory and RDMA-attached storage.
• Assess new runtimes, quantization methods, speculative decoding and mixture-of-experts serving, and recommend which ones to adopt, backed by evidence.
• Help build the orchestration layer around the runtime, covering model admission, GPU scheduling and partitioning, cache-aware request routing and autoscaling.
• Own customer issues from start to finish, find root causes, and improve systems and processes so the same problems do not return.
• Give thoughtful code and design reviews, mentor teammates, and set the standard for engineering practice on the team.

Requirements

• Hands-on familiarity with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI or NVIDIA NIM, including changing engine internals.
• Solid grasp of inference internals: continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization and speculative decoding.
• Working knowledge of tensor and pipeline parallelism, NCCL collectives, and how GPU memory hierarchy and interconnects affect them.
• Advanced Kubernetes skills, including operators, custom resources, controllers and scheduling.
• Strong Go and Python programming. You can read, debug and profile C++/CUDA code with tools such as Nsight.
• Experience debugging and profiling multi-tier applications such as RAG pipelines and agents.
• Strong analytical, debugging and problem-solving skills.
• At least 8 years of software engineering experience, including 1 to 2+ years working directly on LLM inference runtimes or production model serving.
• A degree in Computer Science or a related field.
• Preferred: upstream contributions to vLLM, SGLang, TensorRT-LLM, llm-d, LMCache or KServe.
• Preferred: experience with disaggregated prefill and decode serving, or with KV cache offload and reuse at scale.
• Preferred: experience with RDMA, GPUDirect Storage, InfiniBand or RoCE.
• Preferred: experience with MIG, fractional GPU allocation and multi-tenant GPU isolation.
• Preferred: experience delivering on-premises, air-gapped or regulated enterprise software.

Conditions

Hybrid role: on average 2 days per week in a company office.

The primary work location is the one listed, but the role may be based at any other company site in the US. Remote arrangements will be considered.

US base salary range: USD 144,000 to 273,000 in Colorado, and USD 137,000 to 315,000 in North Carolina and Texas. Variable incentives may also be offered on top of base pay.

Final pay depends on location, experience, education or training, and skill level.

Estimated application deadline: December 1 2026.

The company is an equal opportunity employer that makes decisions on qualifications, merit and business need, and it complies with laws on criminal records. Accommodations are available during hiring.

Benefits

• Health and wellbeing benefits covering physical, financial and emotional needs for employees and their families.
• Personal and professional development programs for reaching career goals, whether that means becoming a subject expert or moving into another division.
• Details of US employee benefits are published on the company's benefits enrollment site.