You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Early Window: Closes in
Company hidden

Member of Technical Staff

  • Office
  • 6+ years

Salary

$250,000 - 400,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company trains and evaluates AI on real professional work for frontier AI labs and enterprises.

Its benchmarks measure how AI performs on work with economic value. Its enterprise evaluations and post-training tools turn those results into measurable gains across real workflows.

Training data today comes from two sources. Human experts such as doctors, lawyers, engineers and analysts write tasks, rubrics and quality judgments. Synthetic pipelines of AI agents build environments, generate tasks and retry until they meet the quality bar. The two sides barely coordinate: experts wait on pipelines and redo work when agents err, pipelines underuse expert judgment, which is the most valuable signal, and no one owns the system connecting them.

This role is for a senior engineer who ships products, understands the infrastructure beneath them, and can reason across product, agent pipelines, research and the business.

Nobody has solved how experts and agents should produce data together, so this is a new product problem. The system could become the base platform for expert work beyond training data, shaping how frontier models are trained and evaluated and the daily experience of thousands of experts and operators.

Responsibilities

  • Design how experts see, steer and correct agent output, and how agents request and use expert input.
  • Build the day-to-day surfaces experts use inside the company's data platform.
  • Build the real-time infrastructure that lets agents act on expert input.
  • Own the metrics: expert time per unit of accepted data, the share of expert input pipelines use, and expert satisfaction and retention.
  • Coordinate with product, synthetic pipelines, research and go-to-market to match customer needs, expert needs and what makes data valuable for training.
  • Set the engineering quality standard and technical direction for the collaboration system.

Requirements

  • 7+ years building and shipping production products that people use.
  • Strong full-stack product skills, a high bar for usability and empathy for end users.
  • Infrastructure experience to build both the front end and the back end, including real-time, high-throughput or pipeline-based systems.
  • Ability to reason about product, systems and the business together, and to recognise where agent pipelines succeed or fail.
  • Strong ownership: comfortable with ambiguity and able to take a project from design through delivery.
  • Clear communication of technical and product trade-offs.
  • Nice to have: experience building human-in-the-loop AI workflows or tools for data labeling, annotation or evaluation.
  • Nice to have: experience building agentic systems or LLM-powered products.
  • Nice to have: familiarity with what makes data valuable for training and evaluating models.
  • Nice to have: experience at top-tier tech companies or fast-scaling startups.

Conditions

Work in person five days a week in San Francisco, New York or London.

The company is an equal opportunity employer and does not discriminate on any legally protected characteristic.

Benefits

  • Equity grant vesting over 4 years.
  • Health, dental and vision insurance.
  • 401(k) with company match.
  • Unlimited paid vacation and sick days.
  • Parental leave for birthing and non-birthing parents.
  • Free mental health services.
  • $200 monthly wellness reimbursement.
  • $1.5K monthly meal stipend.
  • $200 monthly laundry reimbursement.
  • Free Equinox membership.
  • Relocation bonus of up to $15K.

Where you’d work

From the office, 4 or more days a week in the office

The office

Relocation offered

About the company

Company hidden

  • Industry: AI

Office in San Francisco, United States

Your chances

Still hiring, not crowded yet, and you'd be among the first.

  • 19 checks run
  • 8 good signs
  • 2 red flags

Still hiring?

11 checks

Actively hiring

In its favour4

  • Still on the company's own careers site, checked 1 h agoModerate evidence
  • Found in the last 48 hours, before the big job boardsModerate evidence
  • Specific about the basics: pay, place, level, stack and contract all statedSlight evidence
1 moreFewer
  • States its salarySlight evidence

How crowded?

8 checks

Low

In its favour4

  • In its Early Window: not on the big job boards yetStrong evidence
  • In the office in San Francisco: only people nearby can take itSlight evidence
  • In the office 4 or more days a week: most applicants want lessSlight evidence
1 moreFewer
  • Senior level: far fewer people qualifySlight evidence

Against it2

  • Offers relocation: people from other countries apply tooSlight evidence
  • Pays more than 98% of similar roles: that draws applicantsSlight evidence

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details15 facts · Role, Location, Compensation, Employment, Company
Tech stack
  • LLMs
Seniority
Staff
Type
Full-time
Equity
Equity offered
Industry
AI
Specialty
Fullstack
Show 9 more factsShow less

Role

Category
Development
Specialty
Fullstack
Seniority
Staff
Experience
7+ years
Tech stack
  • LLMs

Location

Work model
Office
Region
United States
Office
  • San Francisco, United States
Days in the office
4 or more days
Relocation
Offered

Compensation

Salary
$250,000 - 400,000 / year
Pay period
Annual
Equity
Equity offered

Employment

Type
Full-time

Company

Industry
AI

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason