Senior Network Production Engineer, AI Supercomputing

Description

The company builds and operates very large AI supercomputers that train leading models. The backend network is central to that machine: one faulty NIC, switch, link or path can slow training or break jobs spanning thousands of GPUs.

This is a senior, hands-on role owning the production health and daily operation of the multi-rail Ethernet backend, built on MRC and RDMA. You combine network engineering, distributed systems, hardware health and production operations to keep performance predictable at scale.

The goal is for researchers never to notice the network: large jobs should succeed, performance loss should stay small, and failures should be recovered quickly.

Responsibilities

• Keep the MRC Ethernet backend network available, performant and ready for operations.
• Define and run service-level indicators covering packet loss, congestion, link health, path diversity, bandwidth, tail latency, collective performance and job impact.
• Spot slow or degraded components before they cause training failures or lower model FLOPs utilization.
• Build health models that connect switch, NIC, host, topology and application telemetry.
• Lead responses to high-severity network incidents and take part in the on-call rotation.
• Diagnose faults across GPUs, NICs, switches, cables, firmware, drivers, network operating systems, MRC, NCCL, Kubernetes and training workloads.
• Build automated mitigations such as path avoidance, node quarantine, workload relocation and safe component remediation.
• Write procedures for maintenance, upgrades, rollback, capacity growth and topology changes.
• Work with training teams on collective slowdowns, stalls, timeouts, job restarts and unexplained MFU loss.
• Turn low-level network signals into a clear picture of impact on jobs.
• Set qualification gates that nodes and racks must pass before joining large training pools.
• Run failure-injection and resilience tests covering link, NIC, switch and path failures.
• Improve placement and routing policies for jobs that span racks, rows and network planes.
• Write production-grade software for validation, monitoring, diagnosis, remediation and fleet-wide change management.
• Replace manual investigation with repeatable workflows and actionable alerts.
• Automate compliance checks for firmware, drivers, configuration and network operating systems.
• Maintain accurate inventory, topology and configuration records for the network.
• Cut the time needed to detect, isolate, mitigate and permanently fix failures.
• Lead investigations with internal infrastructure teams, cloud and datacenter operators, Azure Networking, NVIDIA, switch and NIC vendors and network software partners.
• Own incidents through to resolution, including failures that cross team or company boundaries.
• Write root-cause analyses with corrective actions, owners and due dates.
• Turn recurring production failures into requirements for future network, server and accelerator designs.
• Write runbooks and train engineers across regions to provide reliable round-the-clock coverage.

Requirements

• A bachelor's degree in computer science or a related technical field and 4+ years in site reliability engineering, DevOps or infrastructure engineering, or equivalent experience.
• Hands-on experience running large datacenter or high-performance computing networks in production.
• Working knowledge of Ethernet, RDMA, RoCEv2, congestion control, routing, load balancing and lossless or near-lossless network design.
• Experience finding faults across switches, NICs, hosts, drivers, firmware and distributed applications.
• Experience with Linux and with Python, Go, C++ or another systems-oriented language.
• Experience building monitoring, automation, diagnostic or remediation systems for production infrastructure.
• Experience leading high-severity incident response across several engineering teams.
• Experience using telemetry to identify and confirm the root cause of infrastructure or network problems.
• Ability to join a global on-call rotation.
• Preferred: a master's degree in computer science or a related technical field and 2+ years in site reliability engineering, DevOps or infrastructure engineering, or equivalent experience.
• Preferred: experience running GPU training clusters or other tightly coupled distributed systems.
• Preferred: knowledge of MRC or other multi-rail, multi-plane, multipath Ethernet transports.
• Preferred: experience with NCCL, collective communications, CUDA, GPUDirect RDMA and distributed training frameworks.
• Preferred: familiarity with ConnectX-class NICs, modern Ethernet switch ASICs, SONiC, SAI and switch telemetry.
• Preferred: experience with InfiniBand and how its operation differs from Ethernet-based AI fabrics.
• Preferred: familiarity with Kubernetes, Slurm, workload placement and cluster-health certification.
• Preferred: experience with streaming telemetry, gNMI, Prometheus, Datadog, Kusto, eBPF or similar observability tools.
• Preferred: experience managing fleet-wide firmware, driver or network operating-system changes.
• Preferred: a record of improving reliability through automation instead of repeated manual work.

Conditions

Software Engineering IC4 base pay across the US is USD $119,800 - $234,700 per year.

Software Engineering IC4 base pay in the San Francisco Bay Area and New York City metro area is USD $160,200 - $261,000 per year.

Software Engineering IC5 base pay across the US is USD $142,800 - $274,800 per year.

Software Engineering IC5 base pay in the San Francisco Bay Area and New York City metro area is USD $188,000 - $304,200 per year.

The role stays open for at least 5 days and accepts applications until it is filled.

Benefits

• Some roles may be eligible for additional benefits and other compensation; details are available separately.