You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Early Window: Closes in
Company hidden

Principal Engineer Capacity/Compute

  • Hybrid
  • 6+ years

Salary

$320,000 - 485,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company runs one of the largest and fastest-growing infrastructure fleets in the industry, spanning multiple accelerator families, CPU families and cloud providers.

The Capacity Engineering team makes sure every infrastructure resource is accounted for, well utilized and efficiently allocated. It owns the data, tooling and operational systems used to plan, measure and maximize utilization across first-party and third-party compute.

You will build the production systems behind this work: data pipelines that ingest and normalize telemetry from varied cloud environments, observability tooling for real-time fleet health, and instrumentation that measures how efficiently each major workload uses its hardware. You will write production-quality code daily and work with Kubernetes-native infrastructure at significant scale. Your decisions will shape one of the company's largest areas of spend.

You will work closely with research engineering, infrastructure, inference and finance teams. The role suits someone comfortable moving between data engineering, systems engineering and observability, who does well with high autonomy and high ambiguity.

This is a pipeline role feeding four areas. You will focus mainly on one, based on your background and business priorities, but the areas overlap and the boundaries are fluid.

Data platform: pipelines that ingest occupancy and utilization telemetry from Kubernetes clusters, normalize billing and usage across cloud providers, and serve BigQuery tables to the whole organization. Correctness, completeness and latency are central. Consumers range from research engineers to finance and leadership, so the work includes defining schema contracts, making data easy to find, and working out what people need.

Planning: knowing what the fleet has, where it is heading and what blocks it. This covers real-time fleet health tooling, capacity planning platforms, alerts on occupancy drops and allocation problems, and fixes to scheduling and fragmentation. It involves Kubernetes operations on one side and cross-team coordination on the other.

Efficiency: measuring how well each major workload uses its hardware. This means instrumenting utilization across training, inference and eval systems, building benchmarking infrastructure, setting per-configuration baselines, and working directly with the teams that own those systems to close gaps. The metric must be credible enough that the owning team accepts the number.

Attribution and forecasting: linking fleet cost to what the business does with it. This includes reconciling cloud billing exports against vendor telemetry and internal systems with mismatched schemas, assigning spend to the workloads and teams behind it, and turning inference demand and research roadmaps into a defensible compute plan. Efficiency metrics must hold up with finance: free of pure demand and unit-price effects, reproducible month to month, and clear to a CFO.

Responsibilities

  • Build the planning and allocation stack: the tools leadership uses to allocate capacity, teams use to plan against their allocations, and the scheduler enforces. This covers cross-region and cross-provider placement, guardrails, queueing and occupancy KPIs.
  • Run efficiency programs covering stranded capacity, rightsizing, recovery of unused capacity, and job-level utilization across training, inference and eval. Set per-configuration baselines and work with system owners to close gaps.
  • Own attribution and forecasting. Reconcile billing from more than ten providers against telemetry and internal systems, assign spend to the workloads producing it, and turn demand signals and research roadmaps into a defensible compute plan and supply pipeline.
  • Build the data platform beneath all of this: pipelines that bring occupancy, utilization and cost from a fast-diversifying fleet into BigQuery. You own completeness, latency SLOs and gap detection. Each new provider is a new integration.
  • Operate Kubernetes-native systems at scale, including collection agents, workload labeling, and the taints, reservations and scheduling behavior that determines which capacity is actually usable.
  • Treat the output as a product. Gather your own requirements, define schema contracts, and design for consumers from research engineers to a CFO. You also own on-call and SLOs, since these surfaces are critical to the company.

Requirements

  • A strong record building and running production systems. This is hands-on engineering with a devops emphasis.
  • Production-quality Python and SQL. Most pipeline code is Python, and the presentation layer is BigQuery SQL, including table-valued functions and views. Both must be idiomatic, well tested and maintainable.
  • Deep experience with at least one major cloud provider (Amazon Web Services, Google Cloud or Microsoft Azure) and its operations.
  • Experience with Prometheus, PromQL and Grafana, including writing recording rules and building monitoring that engineering teams depend on.
  • Ability to gather your own requirements, work across organizational boundaries, and make progress in an ambiguous setting with limited direction.
  • A bachelor's degree or equivalent combination of education, training and/or experience, in a field relevant to the role as shown through coursework, training or professional experience. Years of experience required follow the internal level for the position.
  • Preferred: experience with capacity planning, resource management or cost attribution at a hyperscaler or in a large machine learning environment. Product engineering and developer experience count toward this.
  • Preferred: scheduling and packing efficiency, or profiling-driven optimization of large distributed workloads.
  • Preferred: multi-cloud data ingestion, especially normalizing billing exports, reservation APIs, on-demand capacity reservations, commitments and vendor telemetry from providers with different billing arrangements.
  • Preferred: total cost of ownership and forecasting, including determining whether infrastructure growth is caused by or merely correlated with business drivers.
  • Preferred: accelerator infrastructure familiarity, such as GPU metrics (DCGM), TPU utilization, Trainium power and utilization metrics, or machine learning training and inference systems at the hardware level.
  • Preferred: building internal data products with self-service access, schema contracts, API serving, documentation and discoverability, with attention to how the data is consumed.
  • Preferred: storage efficiency, retention and lifecycle programs at exabyte scale.

Conditions

Annual salary range: $320,000 to $485,000 USD.

Location-based hybrid policy: staff are currently expected to be in an office at least 25% of the time. Some roles may require more office time.

Visa sponsorship is available, but not for every role or every candidate. If an offer is made, the company will make every reasonable effort to obtain a visa and retains an immigration lawyer to help.

Candidates are encouraged to apply even if they do not meet every qualification. Equal-opportunity statement: the company seeks a range of diverse perspectives on its team.

Benefits

  • Competitive compensation and benefits, optional equity donation matching, generous vacation and parental leave, and flexible working hours.
  • Office space for collaborating with colleagues in San Francisco.

Where you’d work

Part of the week in the office

The office

Visa sponsored

About the company

Company hidden

  • Industry: AI

Office in San Francisco, United States

Your chances

Still hiring, not crowded yet, and you'd be among the first.

  • 18 checks run
  • 7 good signs
  • 3 red flags

Still hiring?

11 checks

Actively hiring

In its favour4

  • Still on the company's own careers site, checked 1 h agoModerate evidence
  • Found in the last 48 hours, before the big job boardsModerate evidence
  • Specific about the basics: pay, place, level, stack and contract all statedSlight evidence
1 moreFewer
  • States its salarySlight evidence

Against it1

  • Listed for the 4th time after earlier listings of it closedModerate evidence

How crowded?

7 checks

Low

In its favour3

  • In its Early Window: not on the big job boards yetStrong evidence
  • Hybrid in San Francisco: only people nearby can take itSlight evidence
  • Senior level: far fewer people qualifySlight evidence

Against it2

  • Sponsors visas: applicants from abroad compete for it tooSlight evidence
  • Pays more than 98% of similar roles: that draws applicantsSlight evidence

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details14 facts · Role, Location, Compensation, Employment, Company
Tech stack
  • Python
  • SQL
  • AWS
  • Google Cloud
  • Kubernetes
  • Azure
Seniority
Principal
Type
Full-time
Equity
Equity offered
Industry
AI
Specialty
Backend
Show 8 more factsShow less

Role

Category
Development
Specialty
Backend
Seniority
Principal
Experience
6+ years
Tech stack
  • Python
  • SQL
  • AWS
  • Google Cloud
  • Kubernetes
  • Azure

Location

Work model
Hybrid
Region
United States
Office
  • San Francisco, United States
Visa sponsorship
Sponsored

Compensation

Salary
$320,000 - 485,000 / year
Pay period
Annual
Equity
Equity offered

Employment

Type
Full-time

Company

Industry
AI

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason