DSX NVCRE Senior Engineer

Description

The company is hiring a Senior Software Engineer for its DSX team to build and run the tooling that confirms AI infrastructure is ready for production.

The work covers cluster readiness and certification software, including the open-source Cluster Readiness Engine. It checks GPU clusters running Kubernetes and Slurm, measures real training and inference performance, and produces evidence that customers and partners can rely on.

The team works with leading AI companies and cloud providers to confirm their infrastructure performs as designed before their most important workloads run on it.

The role is for an existing vacancy.

Responsibilities

• Design and build software that turns cluster validation from a manual task into repeatable, verifiable processes, shaping how the company and its partners qualify new GPU platforms and deployments.
• Design and implement readiness and certification tools in Go and Python for Kubernetes and Slurm, including controllers, operators, CLIs and scheduler integrations.
• Build a catalog of benchmarks and real workloads covering communication (NCCL), compute, storage, diagnostics, training, inference, reliability and security, with expectations for each architecture and clear pass/fail criteria.
• Build provenance and evidence pipelines that record what ran, where it ran and under which configuration, producing signed, auditable results.
• Bring new GPU platforms, schedulers and fabrics under test, from first bring-up on a single node to multi-node validation on production clusters.
• Work with internal platform, networking and solutions teams and cloud partners to find the root cause of performance and correctness gaps, and send fixes back into platforms and reference architectures.
• Contribute to the open-source Cluster Readiness Engine and related projects through design documents, documentation and code reviews.
• Write design documents, runbooks and operational guides that explain how clusters are validated and how results should be read.

Requirements

• A BS, MS or PhD in computer science, computer or electrical engineering, or a related technical field, or equivalent experience.
• 8+ years of software engineering experience building infrastructure or systems software, ideally for large-scale compute, cloud or HPC environments.
• Strong Go and Python skills, with experience shipping production-quality services, controllers or command-line tools.
• Deep knowledge of Linux, containers and Kubernetes, including writing operators or controllers against the Kubernetes API.
• Working knowledge of HPC schedulers such as Slurm and of GPU cluster basics: drivers, NCCL, InfiniBand or RoCE fabrics, and GPU health tools such as DCGM.
• Experience benchmarking or validating distributed systems with a disciplined approach to measurement, reproducibility and evidence.
• A proven ability to investigate complex technical problems to root cause and explain findings clearly to engineers and leadership.
• Nice to have: hands-on experience running or tuning large-scale distributed training or inference workloads with frameworks such as PyTorch, Megatron, NeMo, vLLM, SGLang or TensorRT-LLM.
• Nice to have: experience with the company ecosystem, including DGX systems, GB200/GB300 platforms, CUDA, NVLink and NVSwitch fabrics, and the company networking.
• Nice to have: prior contributions to open-source infrastructure projects, especially Kubernetes operators, scheduler plugins or benchmarking frameworks.
• Nice to have: background in software supply-chain security, attestation or signed artifacts, and in building audit-grade evidence into systems.
• Nice to have: familiarity with observability stacks (Prometheus, Grafana, OpenTelemetry), GitOps workflows and infrastructure as code for GPU clusters.

Conditions

Applications are accepted at least until November 1, 2026.

Base salary depends on location, experience and the pay of employees in similar roles. The base salary range is 184,000 USD to 287,500 USD for Level 4, and 224,000 USD to 356,500 USD for Level 5.

The company uses AI tools in its recruiting process.

The company is an equal opportunity employer and does not discriminate on the basis of any characteristic protected by law.