You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
NVIDIA

Senior Software Engineer, AI Compute Scheduling

  • Office
  • 6+ years

Salary

$184,000 - 287,500/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company invented the GPU and works across gaming, graphics, high-performance computing and AI, with technology used in generative AI and autonomous systems.

The MARS Scheduling team decides how the company's AI research compute is allocated. The fleet includes many of the largest GPU clusters in the world, spread across several clouds and partners, and runs pretraining, reinforcement learning and agentic workloads.

The team runs Slurm and Kubernetes at large scale, with topology-aware scheduling on rack-scale systems, fair sharing and quotas for hundreds of research teams, preemption, reservations, and elastic jobs that resume after failures without losing progress.

You will own parts of the scheduling stack, contribute to open-source and internal platform code, and see your work return GPU-hours to researchers within days. The researchers training the company's own models are your users and will give direct feedback.

Responsibilities

  • Design and build scheduling for large GPU fleets on Slurm and Kubernetes, including queueing, gang and topology-aware placement on rack-scale NVLink systems, preemption, reservations, and backfill for very deep queues.
  • Build the policy layer for fair share, quotas and allocations across many research teams, driven by a central allocation system and enforced the same way on every scheduler.
  • Run Slurm as a cloud-native service on Kubernetes and extend Kubernetes batch scheduling with Kueue and its ecosystem for training, reinforcement learning, sandboxes and inference.
  • Build services around the scheduler, such as reservation automation, node-health-aware placement, cross-cluster workflow orchestration, and APIs that let other platforms submit and track work.
  • Measure and improve goodput, including GPU occupancy, waste, fairness and time to start for large jobs, and use that data to decide what to build next.
  • Own reliability for the components you build by staying on call, analysing incidents, and fixing whole classes of failure before researchers run into them.
  • Contribute upstream to Slurm, Kubernetes and related open-source projects, and work with platform, storage, networking and research teams.

Requirements

  • A BS or MS in Computer Science, Computer Engineering, Electrical Engineering or a related field, or equivalent experience.
  • 6+ years building production systems software.
  • Deep experience with at least one batch or cluster scheduler, such as Slurm, Kueue, Volcano, KAI, YuniKorn, or a scheduler you built, including an understanding of its internals and not only its configuration.
  • Strong programming in Go, C or C++, plus Python for tooling and analysis.
  • Hands-on Kubernetes experience with controllers or operators, CRDs, and scheduling and resource management.
  • Solid Linux systems knowledge, including containers (Docker, Podman, Enroot or Apptainer), cgroups, basic networking, and debugging under load.
  • A record of owning a production service and making it measurably better.
  • Clear written and spoken communication with engineers and researchers.
  • Ways to stand out: experience scheduling large-scale AI training with thousands of GPUs, NVLink or InfiniBand or RoCE topology, and multi-node jobs that must start together.
  • Ways to stand out: contributions to Slurm, Kubernetes SIG Scheduling, Kueue, Volcano, KAI or similar open-source schedulers.
  • Ways to stand out: experience running Slurm on Kubernetes (Slinky or similar), or building Kubernetes operators for batch or HPC workloads.
  • Ways to stand out: familiarity with RL, agentic or inference workloads and their scheduler needs, such as heterogeneous jobs, sandboxes, elastic scale and fast restarts.
  • Ways to stand out: experience running workflow orchestrators such as Flyte, Airflow, Ray or Kubeflow at scale.

Conditions

Base salary depends on location, experience and other objective job-related factors. The expected base salary is 184,000 USD to 287,500 USD for Level 4, and 224,000 USD to 356,500 USD for Level 5.

This is an existing vacancy. Applications will be accepted at least until November 14, 2026.

The company uses AI tools in its recruiting process.

The company is an equal opportunity employer and does not discriminate on the basis of any characteristic protected by law.

Where you’d work

From the office

About the company

NVIDIA

Offices in United States, Austin, United States

Also hiring in Redmond, United States, Seattle, United States, Helsinki, Finland and 13 more places

163 of their 483 open roles are remote

Your chances

Worth a look before you spend an evening tailoring a CV for it.

  • 21 checks run
  • 1 red flag

Still hiring?

14 checks

1 red flag

How crowded?

7 checks

No red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details12 facts · Role, Location, Compensation, Employment
Tech stack
  • Python
  • Go
  • Kubernetes
  • Docker
  • C++
  • Linux
  • LLMs
Seniority
Senior
Type
Full-time
Equity
Equity offered
Specialty
Backend
Region
United States
Show 6 more factsShow less

Role

Category
Development
Specialty
Backend
Seniority
Senior
Experience
6+ years
Tech stack
  • Python
  • Go
  • Kubernetes
  • Docker
  • C++
  • Linux
  • LLMs

Location

Work model
Office
Region
United States
Offices
  • United States
  • Austin, United States

Compensation

Salary
$184,000 - 287,500 / year
Pay period
Annual
Equity
Equity offered

Employment

Type
Full-time

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason