The company trains and evaluates AI on real professional work for frontier AI labs and enterprises.
Its benchmarks measure how AI performs on work with economic value. Its enterprise evaluations and post-training tools turn those results into measurable gains across real workflows.
Training data today comes from two sources. Human experts such as doctors, lawyers, engineers and analysts write tasks, rubrics and quality judgments. Synthetic pipelines of AI agents build environments, generate tasks and retry until they meet the quality bar. The two sides barely coordinate: experts wait on pipelines and redo work when agents err, pipelines underuse expert judgment, which is the most valuable signal, and no one owns the system connecting them.
This role is for a senior engineer who ships products, understands the infrastructure beneath them, and can reason across product, agent pipelines, research and the business.
Nobody has solved how experts and agents should produce data together, so this is a new product problem. The system could become the base platform for expert work beyond training data, shaping how frontier models are trained and evaluated and the daily experience of thousands of experts and operators.
Responsibilities
Design how experts see, steer and correct agent output, and how agents request and use expert input.
Build the day-to-day surfaces experts use inside the company's data platform.
Build the real-time infrastructure that lets agents act on expert input.
Own the metrics: expert time per unit of accepted data, the share of expert input pipelines use, and expert satisfaction and retention.
Coordinate with product, synthetic pipelines, research and go-to-market to match customer needs, expert needs and what makes data valuable for training.
Set the engineering quality standard and technical direction for the collaboration system.
Requirements
7+ years building and shipping production products that people use.
Strong full-stack product skills, a high bar for usability and empathy for end users.
Infrastructure experience to build both the front end and the back end, including real-time, high-throughput or pipeline-based systems.
Ability to reason about product, systems and the business together, and to recognise where agent pipelines succeed or fail.
Strong ownership: comfortable with ambiguity and able to take a project from design through delivery.
Clear communication of technical and product trade-offs.
Nice to have: experience building human-in-the-loop AI workflows or tools for data labeling, annotation or evaluation.
Nice to have: experience building agentic systems or LLM-powered products.
Nice to have: familiarity with what makes data valuable for training and evaluating models.
Nice to have: experience at top-tier tech companies or fast-scaling startups.
Conditions
Work in person five days a week in San Francisco, New York or London.
The company is an equal opportunity employer and does not discriminate on any legally protected characteristic.
Benefits
Equity grant vesting over 4 years.
Health, dental and vision insurance.
401(k) with company match.
Unlimited paid vacation and sick days.
Parental leave for birthing and non-birthing parents.
Free mental health services.
$200 monthly wellness reimbursement.
$1.5K monthly meal stipend.
$200 monthly laundry reimbursement.
Free Equinox membership.
Relocation bonus of up to $15K.
Where you’d work
From the office, 4 or more days a week in the office