Software Engineer, ML Performance

Description

The company has built GPUs for graphics, PC gaming and accelerated computing for over 25 years, and now applies that work to AI, where the GPU drives computers, robots and self-driving cars.

The team is based in Santa Clara, CA, and this role focuses on the performance of AI systems at very large scale.

Responsibilities

• You will analyze the speed of distributed training and inference jobs across every layer, from model and framework through data loading, checkpointing, compiler and kernels, CUDA and NCCL, down to host and network. For each bottleneck, you will fix it or propose a fix and back it with evidence.
• You will decide which performance metrics to track across a GPU fleet and split each one into the factors engineers need when debugging. You will also choose which signals the platform collects first.
• You will lead regression analysis by splitting workloads across machines, software versions and configurations to find the cause, then package the method so others can reuse it.
• You will check automated diagnoses for accuracy, spot misleading results and missed problems, and build labeled test cases that keep the analysis reliable.
• You will work directly with partners on their real workloads, turn findings into improvements, and pass gaps in the product back to the roadmap.

Requirements

• You have a bachelor's degree in computer science, computer engineering or a related field, or equivalent experience.
• You have 12+ years of software engineering experience, including hands-on work optimizing GPU training or inference workloads in production.
• You have done performance work across more than one layer of the stack and can show the baseline, the evidence, the alternative explanations and the validated result.
• You have diagnosed performance across many jobs, machines or tenants, not just one workload.
• You know framework, runtime and compiler interfaces well enough to say where a signal belongs, even when someone else writes the instrumentation.
• You can separate symptoms from causes, recognize noisy or incomplete measurements and the distortion caused by measuring, and tell when the evidence is not enough.
• You write maintainable analysis code, tests or repeatable investigation procedures, and you debug well, including when using automated development tools.
• You communicate findings, trade-offs and remaining uncertainty clearly to the people who own the workload.
• You are proficient in Python and CUDA.
• It would set you apart to have improved both training and inference, to know one or more of PyTorch, JAX, Megatron, NCCL, TensorRT-LLM, vLLM, SGLang or Dynamo in depth, to have moved between application, system software, driver or hardware roles, to apply statistics to noisy multi-node fleet data (counterfactual analysis and workload segmentation are a plus), or to have built debugging or analysis tools that other engineers used and that made their investigations faster.

Conditions

Base salary depends on location, experience and the pay of employees in similar roles. The base salary range is 224,000 USD - 356,500 USD.

Applications are accepted at least until November 2, 2026.

This posting is for an existing vacancy.

The company uses AI tools in its recruiting process.

The company is an equal opportunity employer and does not discriminate on protected characteristics.