Senior Network Production Engineer, AI Supercomputing

Description

The company builds and runs very large AI supercomputers that train leading-edge models.

The backend network is central to that machine. One faulty NIC, switch, link or path can slow training or break jobs that span thousands of GPUs.

This role owns the daily production health and operation of the multi-rail Ethernet backend, which runs on MRC and RDMA. You will connect network engineering, distributed systems, hardware health and operations so the network stays predictable at very large scale.

The goal is that researchers never notice the network: large jobs succeed, performance holds, and failures get fixed quickly.

Responsibilities

• Keep the MRC Ethernet backend healthy, available and ready for operations.
• Set and track service-level indicators for loss, congestion, link state, path diversity, bandwidth, tail latency, collective performance and effect on jobs.
• Spot slow or failing parts early, before they cause training failures or lower model FLOPs utilization.
• Build health models that link switch, NIC, host, topology and application telemetry.
• Join the on-call schedule and run the response when a major network incident happens.
• Find root causes across GPUs, NICs, switches, cables, firmware, drivers, network OS, MRC, NCCL, Kubernetes and training workloads.
• Build automated mitigations such as routing around bad paths, quarantining nodes, moving workloads and safely remediating components.
• Write procedures for maintenance, upgrades, rollback, capacity growth and topology changes.
• Work with training teams on collective slowdowns, stalls, timeouts, job restarts and unexplained MFU loss.
• Explain low-level network signals in terms of their effect on a job.
• Define qualification gates that nodes and racks must pass before joining large training pools.
• Run failure-injection and resilience tests for link, NIC, switch and path failures.
• Improve placement and routing policies for jobs across racks, rows and network planes.
• Write production-grade software for validation, monitoring, diagnosis, remediation and fleet change management.
• Replace manual investigation with repeatable workflows and actionable alerts.
• Automate compliance checks for firmware, drivers, configuration and network OS.
• Keep accurate records of network inventory, topology and configuration.
• Shorten the time it takes to detect, isolate, mitigate and permanently fix failures.
• Lead investigations with internal infrastructure teams, cloud and datacenter operators, Azure Networking, NVIDIA, switch and NIC vendors and network software partners.
• Own incidents through to resolution, including those that cross team boundaries.
• Write root-cause analyses with corrective actions, owners and dates.
• Turn repeated production failures into requirements for future network, server and accelerator designs.
• Write runbooks and train engineers in each region so coverage runs around the clock.

Requirements

• Either a Bachelor's degree in computer science or a related technical discipline plus 4+ years of technical experience in site reliability, DevOps or infrastructure engineering, or equivalent experience.
• Hands-on experience running large datacenter or HPC networks in production.
• Experience with Ethernet, RDMA, RoCEv2, congestion control, routing, load balancing and lossless or near-lossless design.
• Experience diagnosing faults across switches, NICs, hosts, drivers, firmware and distributed applications.
• Experience with Linux and programming in Python, Go, C++ or another systems-oriented language.
• Experience building monitoring, automation, diagnostic or remediation tools for production.
• Experience leading high-severity incident response across several engineering teams.
• Experience using telemetry to confirm root causes of infrastructure or network problems.
• Ability to join a global on-call rotation.
• Preferred: a Master's degree in computer science or a related technical discipline plus 2+ years of technical experience in site reliability, DevOps or infrastructure engineering, or equivalent experience.
• Preferred: experience operating GPU training clusters or other tightly coupled distributed-computing systems.
• Preferred: knowledge of MRC or other multi-rail, multi-plane, multipath Ethernet transports.
• Preferred: experience with NCCL, collective communications, CUDA, GPUDirect RDMA and distributed training frameworks.
• Preferred: familiarity with ConnectX-class NICs, modern Ethernet switch ASICs, SONiC, SAI and switch telemetry.
• Preferred: experience with InfiniBand and how it differs operationally from Ethernet-based AI fabrics.
• Preferred: familiarity with Kubernetes, Slurm, workload placement and cluster-health certification.
• Preferred: experience with streaming telemetry, gNMI, Prometheus, Datadog, Kusto, eBPF or equivalent observability systems.
• Preferred: experience managing fleet-wide firmware, driver or network operating-system changes.
• Preferred: a record of improving reliability through automation instead of recurring manual work.

Conditions

Software Engineering IC5 base pay across the United Kingdom is typically £93,500.00 to £161,800.00 per year.

The role stays open for a minimum of 5 days, and applications are accepted on an ongoing basis until it is filled.

Benefits

• Some roles may qualify for benefits and other compensation.
• Additional benefits and pay details are available from the company.