Principal Software Engineer – Large-Scale LLM Memory and Storage Systems

Description

The company builds Dynamo, an inference framework that serves generative AI and reasoning models across many machines with high throughput and low latency. It is written in Rust with Python around it, and it schedules GPU shards, routes requests and shares a KV cache across mixed hardware clusters so the accelerators work like one datacenter-scale system.

As language models outgrow the memory and compute of a single GPU, the platform aims to make deploying them efficient and resilient.

The team wants a Principal Systems Engineer to chart direction and plan for how memory is managed across large LLM and storage platforms. Engineering teams here are growing quickly.

Responsibilities

• Design and grow one memory layer that covers GPU memory, pinned host memory, RDMA-reachable memory, SSD tiers, and remote file, object and cloud storage for LLM inference.
• Build and own deep integrations with major serving engines such as vLLM, SGLang and TensorRT-LLM, focused on offloading, reusing and remotely sharing KV cache across mixed and disaggregated clusters.
• Define interfaces and protocols for disaggregated prefill, peer-to-peer KV cache sharing, and KV cache storage across GPU, CPU, local disk and remote memory tiers.
• Work with GPU architecture, networking and platform teams to use GPUDirect, RDMA, NVLink and similar technologies for fast KV cache access across accelerators and memory pools.
• Mentor senior and junior engineers, set technical direction for memory and storage subsystems, and speak for the team in internal reviews and in external settings such as open source, conferences and customer technical deep dives.

Requirements

• Masters or PhD, or equivalent experience.
• 15+ years working on large-scale distributed systems, high-performance storage or ML infrastructure in C/C++ and Python, including production services.
• A solid grasp of memory hierarchies, from GPU HBM and host DRAM to SSD and remote or object storage, and experience designing systems that span these tiers for performance and cost.
• Experience with distributed caching or key-value systems built for low latency and high concurrency.
• Hands-on work with networked I/O and RDMA, NVMe-oF or NVLink-style technologies, plus familiarity with disaggregated and aggregated AI cluster deployments.
• Strong at profiling and tuning across CPU, GPU, memory and network, using metrics to guide architecture and confirm gains in TTFT and throughput.
• Excellent communication, and past experience leading cross-functional work with research, product and customer teams.
• Ways to stand out: prior contributions to open-source LLM serving or systems projects on KV cache optimization, compression, streaming or reuse.
• Ways to stand out: experience designing unified memory or storage layers that present one logical KV or object model across GPU, host, SSD and cloud tiers, especially in enterprise or hyperscale settings.
• Ways to stand out: publications or patents on LLM systems, memory-disaggregated architectures, RDMA or NVLink data planes, or CDN-like KV cache systems for ML.

Conditions

Base salary is set by location, experience and other objective, job-related factors, with an expected range of $272,000 to $431,250 USD.

Applications are accepted at least until November 3, 2026.

This is an existing vacancy.

The company uses AI tools in parts of its recruiting process.

The company is an equal opportunity employer and does not discriminate on the basis of protected characteristics.

Benefits

• Highly competitive salary and a comprehensive benefits package.