AI Reliability Engineer, HPC

Description

The company is pushing AI forward and wants people who can take on hard, open problems in building systems with real intelligence across agents, applications, services and infrastructure. Its aim is to make AI usable by consumers, businesses and developers alike.

The team is hiring an experienced AI Reliability Engineer for its High Performance Computing (HPC) infrastructure group. The role combines software and systems engineering to keep large distributed AI infrastructure reliable and efficient, with very high uptime.

The team works with product groups that bring the company's models to billions of users.

Responsibilities

• Keep HPC clusters used for model training and inference available, resilient and fault tolerant.
• Design and run monitoring, alerting and logging so you can see in real time what is happening across GPUs, clusters, storage and networking.
• Build automation for deployments, incident response, scaling and failover across CPU and GPU environments.
• Take part in on-call rotations, troubleshoot production problems, run blameless postmortems and push improvements that follow from them.
• Keep training and serving environments secure and compliant, with data privacy in mind.
• Work with ML engineers and platform teams to improve the developer experience and move research into production faster.
• Live the company's culture and values.

Requirements

• Required Qualifications: a bachelor's degree in computer science or a related technical field and 4+ years of technical experience in site reliability engineering, DevOps or infrastructure engineering, or equivalent experience.
• Preferred Qualifications: a master's degree in computer science or a related technical field and 2+ years of technical experience in site reliability engineering, DevOps or infrastructure engineering, or equivalent experience.
• Preferred: hands-on experience with Kubernetes, Docker, container orchestration and CI/CD pipelines for ML training or inference workloads.
• Preferred: experience with public cloud platforms such as Azure, AWS or GCP, including infrastructure as code.
• Preferred: experience with monitoring and observability tools such as Grafana, Datadog or OpenTelemetry.
• Preferred: scripting or programming in Python, Go or Bash.
• Preferred: experience with distributed systems, networking, storage and high performance computing (HPC).
• Preferred: experience running GPU clusters and workload schedulers for ML and AI workloads.
• Preferred: experience with ML training or inference pipelines.
• Preferred: experience with capacity planning and cost optimization for GPU-based infrastructure.

Conditions

Level: Software Engineering IC5.

The typical base pay range across the United Kingdom is £93,500.00 to £161,800.00 per year.

You are expected to work from a designated Microsoft office at least four days a week if you live within 50 miles (U.S.) or 25 miles (non-U.S., country-specific) of that location. This is subject to local law and may differ by jurisdiction.

The position stays open for at least 5 days, and applications are taken on an ongoing basis until it is filled.

Benefits

• Some roles may be eligible for benefits and other compensation.