Member of Technical Staff (Software Engineer, GPU Cluster Infrastructure)

Description

The company handles hundreds of millions of queries a month, and each query splits into several real-time AI inference requests running on a large GPU fleet spread across multiple cloud providers.

Inference engineers and researchers currently build models while also handling networking, securing capacity, and running the GPU clusters. This team takes over that infrastructure and wraps its complexity in one self-serve platform for training and inference workloads.

Responsibilities

• Design and own a self-serve compute platform that lets inference engineers and researchers start training jobs and run inference services without handling GPU provisioning, cluster setup, or provider-specific details.
• Own provisioning, lifecycle management, reliability, and capacity across providers, so teams use compute the same way wherever it runs.
• Build scheduling and placement logic that finds open capacity across providers, packs it efficiently, and matches each workload to suitable hardware under real limits.
• Keep long-running distributed training jobs healthy while guaranteeing availability and latency for production inference services on the same fleet.
• Write Kubernetes operators and CRDs and manage many clusters across providers so the platform behaves identically everywhere.
• Build fault tolerance, autoscaling, and observability that keep the fleet utilized and let workloads survive node loss, provider issues, and capacity changes without manual intervention.
• Work with inference and cloud infrastructure engineers to turn operational constraints into a platform architecture and roadmap.

Requirements

• The team expects real depth in most of the following areas:
• Deep Kubernetes experience, including custom operators, CRDs, and multi-cluster federation, beyond routine kubectl use.
• Experience managing GPU clusters at scale with NVIDIA hardware, CUDA, and fast networking such as InfiniBand or RoCE.
• Experience orchestrating compute across multiple clouds such as CoreWeave, AWS, or GCP, and an understanding of how much they differ.
• Solid distributed systems knowledge in scheduling, resource allocation, and fault tolerance under load.
• Systems-level coding in Go, Rust, or C++.
• Experience running both long-running training jobs and high-availability inference services, and understanding why they demand opposite things from infrastructure.
• A habit of owning problems end to end, including when no clear path is defined.
• Additional experience the team values:
• Inference serving stacks such as vLLM, SGLang, or TensorRT-LLM (valued).
• Slurm or other HPC schedulers (valued).
• GPU kernel work in CUDA or Triton (not required, but notable).
• High-speed interconnects such as InfiniBand, RoCE, or RDMA in production (valued).
• Observability tools for ML workloads such as Prometheus, Grafana, or Weights & Biases (valued).