AI Infrastructure Engineer

Description

The company organizes human expertise to train and evaluate frontier AI models. It is a profitable Series C business valued at $10 billion, with millions of domain experts on its platform paid over $4 million per day.

The company also runs an enterprise product that helps Fortune 500 firms capture how their top people work and turn that expertise into AI agents.

All of the company's AI work, including training, serving models and building agents, runs on GPUs. This role keeps that compute fast, fault-tolerant and fully used.

You will own the accelerator platform from end to end: clusters and schedulers across AWS, GCP, neoclouds and on-prem, training and LLM serving on top, and the observability and cost data that show where each GPU-hour goes.

This is a software engineering role rather than an operations role. You will write production code daily, own systems from design through on-call, and build the platform that sets the pace of the company's progress.

Representative projects include a multi-cloud GPU fleet managed through custom Kubernetes and Slurm operators, a gang-scheduling and fair-share layer that raises accelerator utilization, a vLLM serving stack with traffic replay benchmarks, a self-healing fault remediation system, a cost-attribution pipeline feeding the warehouse, and internal dashboards used by researchers and non-engineers.

The team works in person five days a week from the San Francisco, NYC or London office.

Responsibilities

• Design, deploy and operate GPU clusters across AWS, GCP and other hyperscalers, neoclouds and on-premise data centers, including custom Kubernetes and Slurm operators and the metadata services behind them.
• Build scheduling that keeps accelerators busy, covering gang scheduling for distributed jobs, bin-packing to limit fragmentation, and quotas and fair share across teams.
• Run distributed training infrastructure on PyTorch Distributed, Ray or Megatron, with fast checkpointing and model loading.
• Deploy and tune high-throughput LLM serving with vLLM or similar tools, including tensor and model or expert parallelism, prefill/decode disaggregation, KV cache offload and FP8/FP4 quantization.
• Build benchmarking and traffic-replay tools that measure TTFT, TPOT and latency percentiles, and set performance baselines for each config.
• Build health checks across hardware, network and software, plus automated remediation that handles fault storms without human intervention.
• Instrument fleet utilization with DCGM, Prometheus and Grafana, and write recording rules and alerts that teams depend on.
• Ingest utilization, occupancy and billing data into a warehouse, attribute spend to teams and workloads, and support capacity planning.
• Lead autoscaling, rightsizing and idle-capacity recovery work with measurable cost results.
• Write clear runbooks and documentation, and explain infrastructure tradeoffs to researchers, finance and leadership.
• Build internal tools and dashboards that researchers and non-engineers use daily.

Requirements

• 5+ years building and operating production distributed systems or ML infrastructure.
• Production-quality Python plus Go or C++, and fluent shell scripting for system administration.
• Deep hands-on Kubernetes and container experience, plus working knowledge of Slurm or another HPC resource manager.
• Experience running GPU workloads at scale, including scheduling, orchestration and troubleshooting.
• Hands-on experience with at least one distributed training or serving stack: PyTorch Distributed, Ray, vLLM or Megatron-LM.
• Deep experience with at least one major cloud provider.
• Familiarity with observability tools such as DataDog, Prometheus, PromQL and Grafana, and comfort being on-call for critical systems.
• Ability to own projects end to end and define your own requirements with little direction.
• Nice to have: LLM inference optimization (KV cache systems, disaggregated serving, quantization, speculative decoding).
• Nice to have: high-performance networking (RDMA, InfiniBand, NCCL, NVLink).
• Nice to have: GPU profiling with Nsight or similar, and CUDA or Triton kernel work.
• Nice to have: experience with accelerators beyond NVIDIA, such as AMD MI-series, TPU or Trainium.
• Nice to have: capacity planning, billing normalization or cost attribution across providers.
• Nice to have: automated fault diagnosis, log mining or AIOps at fleet scale.
• Nice to have: configuration management (Ansible, Puppet) and secure or air-gapped deployments.
• Strong candidates may also have experience with, or a strong desire to learn, the items above.

Conditions

Full-time, in-person work five days a week in the San Francisco, NYC or London office.

Equal-opportunity employer.

Benefits

• Bi-annual performance bonus.
• Generous equity grant vesting over 4 years.
• Relocation bonus of up to $15k.
• $10K housing bonus if you live within 0.5 miles of the office.
• $1.5K monthly stipend for meals.
• Free Equinox membership.
• $200 monthly laundry reimbursement.
• $200 monthly personal wellness reimbursement.
• Health, dental and vision insurance.