You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Early Window: Closes in
Company hidden

Principal Engineer Capacity/Compute

  • Hybrid
  • 6+ years

Salary

$320,000 - 485,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company manages one of the largest and fastest-growing infrastructure fleets in the industry, spanning multiple accelerator families, CPU families and clouds.

The Capacity Engineering team makes sure all infrastructure resources are accounted for, well utilized and efficiently allocated. It owns the data, tooling and operational systems used to plan, measure and maximize utilization across first-party and third-party compute.

This is a pipeline role feeding four areas: data platform, planning, efficiency, and attribution and forecasting. You will focus mainly on one, depending on your background and business priority, but the boundaries overlap.

The data platform area covers pipelines that ingest occupancy and utilization telemetry from Kubernetes clusters, normalize billing and usage across cloud providers, and serve BigQuery tables. Correctness, completeness and latency matter most. Consumers range from research engineers to finance and leadership, so the work includes defining schema contracts and making data discoverable.

The planning area covers knowing what the fleet has, where it is going and what blocks it. It includes real-time cluster health tooling, capacity planning platforms, alerts on occupancy drops and allocation problems, and fixes to scheduling and fragmentation.

The efficiency area covers measuring how well each major workload uses its hardware across training, inference and eval systems. It includes benchmarking, per-config baselines, and working with system-owning teams to close gaps. The metric must be accurate enough that the owning team agrees with it.

The attribution and forecasting area connects fleet cost to business activity. It covers reconciling cloud billing exports against vendor telemetry and internal systems, attributing spend to workloads and teams, and turning inference demand and research roadmaps into a compute plan. Efficiency metrics must hold up with finance: free of pure demand and unit-price effects, reproducible month to month, and clear to a CFO.

The team works closely with research engineering, infrastructure, inference and finance teams. The environment is high autonomy and high ambiguity.

Responsibilities

  • Build the planning and allocation stack: tools leadership uses to allocate capacity, teams use to plan against their allocations, and the scheduler enforces. This covers cross-region and cross-provider placement, guardrails, queueing and occupancy KPIs.
  • Lead efficiency programs covering stranding, rightsizing, recovery of unused capacity, and job-level utilization across training, inference and eval. Set per-config baselines and work with system-owning teams to close gaps.
  • Own attribution and forecasting. Reconcile billing across more than ten providers against telemetry and internal systems, attribute spend to the workloads that create it, and turn demand signals and research roadmaps into a defensible compute plan and supply pipeline.
  • Build the data platform that ingests occupancy, utilization and cost from a fast-changing fleet into BigQuery. You own completeness, latency SLOs and gap detection. Each new provider is a new integration.
  • Operate Kubernetes-native systems at scale, including collection agents, workload labeling, and the taint, reservation and scheduling behavior that determines which capacity is usable.
  • Treat the output as a product. Gather your own requirements, define schema contracts, and design for consumers from research engineers to a CFO. You own on-call and SLOs for these systems.

Requirements

  • A strong track record building and operating production systems. This is a hands-on engineering role with a devops focus.
  • Production-quality Python and SQL. Most pipeline code is Python. The presentation layer is BigQuery SQL, including table-valued functions and views. Code must be idiomatic, well tested and maintainable.
  • Deep experience with at least one major cloud provider (Amazon Web Services, Google Cloud or Microsoft Azure) and its operations.
  • Experience with Prometheus, PromQL and Grafana, including writing recording rules and building monitoring that engineering teams rely on.
  • Ability to gather your own requirements and work across organizational boundaries in an ambiguous environment with limited direction.
  • Preferred: experience with capacity planning, resource management or cost attribution at a hyperscaler or in a large-scale machine learning environment. Product engineering and developer experience work counts.
  • Preferred: scheduling and packing efficiency, or profiling-driven optimization of large distributed workloads.
  • Preferred: multi-cloud data ingestion, including normalizing billing exports, reservation APIs, on-demand capacity reservations, commitments and vendor telemetry from providers with different billing arrangements.
  • Preferred: total cost of ownership and forecasting, including deciding whether infrastructure growth is driven by or merely correlated with business drivers.
  • Preferred: accelerator infrastructure familiarity, such as GPU metrics (DCGM), TPU utilization, Trainium power and utilization metrics, or machine learning training and inference systems at the hardware level.
  • Preferred: building internal data products with self-service access, schema contracts, API serving, documentation and discoverability.
  • Preferred: storage efficiency, retention and lifecycle programs at exabyte scale.
  • Minimum education: a Bachelor's degree or equivalent combination of education, training and experience. The field of study should be relevant to the role, shown through coursework, training or professional experience.
  • Years of experience required correlate with the internal job level for the position.

Conditions

Annual salary range: $320,000 to $485,000 USD.

Hybrid policy: staff are expected to be in an office at least 25% of the time. Some roles may require more.

Visa sponsorship is available, but not for every role and every candidate. If an offer is made, the company will make every reasonable effort to obtain a visa and retains an immigration lawyer to help.

Candidates are encouraged to apply even if they do not meet every qualification. Equal opportunity statement: the company values a range of diverse perspectives.

Benefits

  • Competitive compensation and benefits.
  • Optional equity donation matching.
  • Generous vacation and parental leave.
  • Flexible working hours.
  • An office space for collaboration with colleagues.

Where you’d work

Part of the week in the office

Visa sponsored

About the company

Company hidden

  • Industry: AI

Office in Seattle, United States

Your chances

Still hiring, not crowded yet, and you'd be among the first.

  • 18 checks run
  • 7 good signs
  • 3 red flags

Still hiring?

11 checks

Actively hiring

In its favour4

  • Still on the company's own careers site, checked 1 h agoModerate evidence
  • Found in the last 48 hours, before the big job boardsModerate evidence
  • Specific about the basics: pay, place, level, stack and contract all statedSlight evidence
1 moreFewer
  • States its salarySlight evidence

Against it1

  • Listed for the 4th time after earlier listings of it closedModerate evidence

How crowded?

7 checks

Low

In its favour3

  • In its Early Window: not on the big job boards yetStrong evidence
  • Hybrid in Seattle: only people nearby can take itSlight evidence
  • Senior level: far fewer people qualifySlight evidence

Against it2

  • Sponsors visas: applicants from abroad compete for it tooSlight evidence
  • Pays more than 98% of similar roles: that draws applicantsSlight evidence

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details14 facts · Role, Location, Compensation, Employment, Company
Tech stack
  • Python
  • SQL
  • AWS
  • Google Cloud
  • Kubernetes
  • Azure
Seniority
Principal
Type
Full-time
Equity
Equity offered
Industry
AI
Specialty
Backend
Show 8 more factsShow less

Role

Category
Development
Specialty
Backend
Seniority
Principal
Experience
6+ years
Tech stack
  • Python
  • SQL
  • AWS
  • Google Cloud
  • Kubernetes
  • Azure

Location

Work model
Hybrid
Region
United States
Office
  • Seattle, United States
Visa sponsorship
Sponsored

Compensation

Salary
$320,000 - 485,000 / year
Pay period
Annual
Equity
Equity offered

Employment

Type
Full-time

Company

Industry
AI

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason