Principal High-Performance LLM Training Engineer

Description

The company builds GPUs, systems and software that power deep learning and AI, and this role focuses on making large-scale AI training and post-training run faster across that full stack.

You will work on frontier LLM workloads running on thousands of GPUs, improve frameworks such as PyTorch, JAX, NeMo and NeMo RL, and use what you learn from real workloads to inform future GPU, system and software roadmaps.

The role spans training behavior at the application level down to framework internals, CUDA libraries, collective communication, memory, networking and GPU architecture. Success means delivering performance gains yourself, setting technical direction, raising the engineering bar, and shaping decisions across multiple teams.

Responsibilities

• Lead end-to-end performance analysis and tuning of new LLM pre-training and post-training workloads on the newest hardware and software.
• Push workloads toward hardware speed-of-light by finding and removing bottlenecks in compute, memory, communication, scheduling, parallelism strategy, kernels, framework overhead and system scaling.
• Build production-quality software, tools, models, benchmarks and analysis infrastructure that raise training efficiency and developer productivity across the AI software stack.
• Create performance models, workload characterizations and simulation methods that inform future GPU, networking, system and software architecture.
• Act as the technical authority on AI training performance, working with GPU architecture, systems, CUDA libraries, compilers, networking, frameworks, product management and applied AI teams.
• Turn workload findings into concrete hardware and software recommendations and push for changes that improve performance across the AI ecosystem.
• Mentor engineers across the organization and set best practices for large-scale AI performance analysis and optimization.

Requirements

• MS or PhD in computer science, electrical or computer engineering, or a related field, or equivalent experience, with 12+ years of relevant work or research.
• Proven principal-level impact in at least one of: large-scale AI training systems, GPU performance optimization, distributed systems, high-performance computing, ML frameworks, compilers or runtimes, or hardware/software co-design.
• Hands-on experience optimizing large deep learning workloads, especially transformer LLM pre-training, reinforcement learning, fine-tuning or other post-training.
• Strong understanding of GPU and AI accelerator architecture, from single devices up to datacenter-scale systems.
• Experience with data, tensor, pipeline, expert and sequence parallelism, activation checkpointing, mixed precision training, and overlapping communication with computation.
• A solid record of using profiling, tracing, benchmarking and performance modeling tools to find complex bottlenecks and deliver measurable gains.
• Strong communication and leadership skills, with the ability to influence architecture and software decisions across teams without direct authority.

Conditions

Base salary depends on location, experience and other objective job-related factors, with an expected range of 272,000 USD to 431,250 USD.

Applications are accepted at least until November 3, 2026.

The company uses AI tools in its recruiting process.

The company is an equal opportunity employer and does not discriminate on protected characteristics.