This senior staff role covers the GPU software stack on every node: firmware, kernel and NVIDIA driver, CUDA runtime and libraries, container runtime, and how Kubernetes and inference services see and share a GPU. When a GPU fails anywhere in the fleet, you find the cause and build software so it does not happen again.
This is a senior staff individual contributor role for someone who understands GPUs from the driver up and still writes code daily. The work varies. One month you may qualify a new driver and CUDA release against every model family served. The next you might build the GPU part of the acceptance suite each node must pass, a detector that flags failing GPUs from Xid and ECC history before customers notice, the bring-up of a new GPU generation, or a trace of a CUDA error back to firmware, a PCIe link or a kernel update. You work directly with the node operations, managed Kubernetes, inference performance and inference engine teams, and they call you when a problem sits below the framework.
Success looks like this: every GPU in every pod runs a known, tested combination of firmware, driver and CUDA; upgrades roll across the fleet without failed inferences; GPU faults are detected, classified and remediated automatically; and a new GPU generation goes from delivery to serving traffic in days. The company is a remote-first team.
Responsibilities
Own the GPU software baseline: versioned, reproducible sets of kernel, NVIDIA driver, firmware, CUDA runtime and libraries (cuDNN, cuBLAS, NCCL, TensorRT) and container toolkit, tested before release and rolled across the fleet with no downtime.
Qualify every driver and CUDA release, maintain the compatibility matrix across GPU generations, frameworks and model families, and catch regressions in correctness, performance and memory use before production.
Find root causes of GPU failures, including Xid errors, CUDA 'unknown error', cuBLAS handle failures, out-of-memory and fragmentation, and GPUs falling off the bus. Separate hardware, firmware, driver, runtime and application causes and prove which one applies.
Build GPU acceptance and diagnostics: burn-in, DCGM diagnostics, memory and interconnect tests, and real-workload tests that gate each node and rack before serving traffic, plus the reports customers get at handover.
Make GPU health visible and self-healing. Combine telemetry from DCGM, the driver, the kernel log and the BMC into one trusted view, classify faults, and automate drain, reset and replacement.
Own how GPUs reach workloads: the container runtime, device plugins, the GPU Operator, MIG and sharing policies, and tenant isolation, on bare metal and on Kubernetes.
Work with hardware and node operations teams to bring up new GPU generations and pods: first driver and firmware, NVLink and NVSwitch validation, and automation that cuts the next bring-up to days.
Benchmark image, video, audio and LLM inference on each GPU generation and software stack, and feed results into placement, pricing and purchasing.
Work with NVIDIA and upstream projects: file driver and firmware bugs with reproducible cases, follow them through, track what each release changes, and contribute fixes to the open-source tools the team relies on.
Use AI for log and Xid triage, test generation, release qualification and runbook automation, so the team's output grows faster than its headcount.
Requirements
12+ years building and operating systems software, several of them on GPU software running in production at scale.
Deep hands-on knowledge of the NVIDIA stack: the kernel driver and modules, CUDA runtime and libraries, NVML, DCGM and the Container Toolkit, and how their versions interact.
A track record of tracing GPU failures to root cause across hardware, firmware, driver, runtime and application, including Xid errors, ECC and row remapping, PCIe and NVLink errors, and thermal and memory faults.
Strong Linux internals: kernel, device drivers and modules, PCIe, NUMA, IOMMU, cgroups and containers, and tools such as perf, eBPF and ftrace.
Write code daily. Fluent in Python and in at least one systems language (C, C++, Rust or Go), and able to write a test or reproduce a bug in CUDA.
Experience running GPUs on Kubernetes and bare metal, including device plugins, the GPU Operator, MIG, and image builds and provisioning at scale.
Working knowledge of how PyTorch, TensorRT and ONNX Runtime use a GPU: memory allocation, compilation and caches, and multi-GPU execution.
Clear technical writing. Design documents, postmortems and ticket updates must be actionable for someone who was not in the discussion.
Use AI daily to speed up your own work, and have a clear view of how a small team with these tools should operate differently.
Comfortable owning outcomes in a fast-moving, ambiguous setting, choosing the most important problem and being accountable for the result.
Time spent building GPU drivers, CUDA libraries or GPU system software at a GPU vendor, cloud provider or HPC centre.
Experience with Blackwell-generation hardware (B200, GB200/GB300 NVL72), NVLink and NVSwitch topologies, and liquid-cooled racks.
Multi-GPU and multi-node execution with NCCL, GPUDirect, and RDMA over RoCE or InfiniBand.
Firmware qualification across BIOS, BMC, NIC and GPU, and acceptance testing with OEMs and ODMs.
Inference serving internals in PyTorch, TensorRT-LLM, vLLM, SGLang, or custom diffusion and video pipelines, including how they load, schedule and fail.
Experience with non-NVIDIA accelerators (AMD or custom silicon) and their software stacks.
An open-source project you wrote or maintain that people run in production.
Multi-tenant isolation and confidential computing on shared GPU hardware.
Experience running a 24x7 service with published service levels.
Conditions
Remote-first. Core hours are reserved for cooperative work and calls; outside them, your schedule is yours.
Release cycles are fast and intense and are followed by real downtime; the team is expected to rest after major pushes.
Equal opportunity applies.
Benefits
Generous paid time off, including vacation, sick days and public holidays.
Meaningful stock options.
Flexible hours outside core collaboration blocks.
Paid family leave for maternity, paternity and caregiving.
Company retreats twice a year in inspiring locations, plus two in-person planning and celebration meetings each year.