You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
PhysicsX

AI Engineer: Evals

  • Office

Salary

Not stated

Similar roles pay £30K - 155K a year · our estimate

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company builds physics AI for industrial firms and is developing a new simulation software stack that covers the full engineering lifecycle.

Its customers include organisations in aerospace and defence, automotive, semiconductors, materials, and energy and renewables, often on their most demanding problems.

The role sits in the Agentic strategy area and is a hands-on engineering position with reach across the whole organisation.

Responsibilities

  • Set the technical direction for how the company evaluates agents and deliver it in practice.
  • Build shared evaluation frameworks, reference implementations and standard paths on top of the existing tooling.
  • Support product, delivery and customer teams in creating strong eval suites, then turn the approaches that work into repeatable guidance.
  • Create datasets and evaluators that mix deterministic checks, domain expert judgement, model-based grading and human review.
  • Join offline evaluation, regression testing and production telemetry so teams can spot drift and learn from real usage.
  • Define cross-product scorecards and shared benchmarks while teams keep ownership of the agents they ship.
  • Publish credible open benchmarks for agents that carry out real engineering and manufacturing workflows.
  • Work with Research and Post-Training on evaluating models and agents, and own the integrity of cross-product, production and public benchmarks, including sign-off when results back product or external claims.
  • Build capability through reviews, examples, training and direct collaboration so rigorous evaluation becomes part of how teams work, not a last step before release.
  • Advise on how this capability should grow and, over time, help shape the team around it.

Requirements

  • You have built and run evaluation systems for agents or LLM-powered products in production.
  • You can take a vague product goal and turn it into datasets, evaluators, experiments and decision criteria that teams trust.
  • You understand how probabilistic systems fail and apply sound statistics, treating benchmark scores as estimates rather than exact figures.
  • You know when to use deterministic evaluators, domain experts, human review or LLM-as-a-judge, and what each one cannot do.
  • You are a strong engineer who ships usable frameworks and production-grade tooling, not only analysis or notebooks.
  • You have helped several teams adopt a technical practice through hands-on delivery, clear standards and practical coaching.
  • You can work directly with customers and domain experts, learn what correct means in their workflows and capture it without losing important detail.
  • Foundations: evaluating multi-step agents, tool calls, structured outputs and long-running workflows.
  • Foundations: trace-based observability and OpenTelemetry concepts.
  • Foundations: dataset design, sampling, experiment design, regression detection and reproducibility.
  • Foundations: CI and release gates for systems whose behaviour varies from run to run.
  • Foundations: clear technical writing for product, research, engineering, delivery and customer audiences.
  • Foundations: judging which measurements should drive a decision and which are only interesting.
  • Experience with Pydantic Evals, Pydantic Logfire or comparable code-first evaluation and observability tools.
  • Experience evaluating agents in engineering, simulation, design, manufacturing or other expert domains.
  • Experience creating public benchmarks or open-source evaluation tooling.
  • Familiarity with post-training, synthetic data, reward signals or model selection.
  • Experience in regulated, customer-operated or air-gapped environments.

Conditions

Based in the United Kingdom headquarters, with offices in London, New York and Singapore, and a growing presence in the Bay Area.

The company is an equal opportunity employer and encourages applications from people in groups underrepresented in tech.

Diversity and inclusion data is collected only to monitor equal opportunities policy and legal compliance, is kept confidential, is used only in aggregate, and does not affect application outcomes.

Benefits

  • The company sponsors bright women from disadvantaged backgrounds through their university degrees in science and mathematics.

Where you’d work

From the office

About the company

PhysicsX

  • Industry: AI

Office in London, United Kingdom

Also hiring in New York, United States

Your chances

We've checked whether it's still hiring and how crowded it is.

  • 19 checks run
  • 0 red flags

Still hiring?

13 checks

No red flags

How crowded?

6 checks

No red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details9 facts · Role, Location, Compensation, Company
Tech stack
  • Python
  • LLMs
Industry
AI
Specialty
ML
Region
United Kingdom
Pay period
Annual
Show 4 more factsShow less

Role

Category
Data & Analytics
Specialty
ML
Tech stack
  • Python
  • LLMs

Location

Work model
Office
Region
United Kingdom
Office
  • London, United Kingdom

Compensation

Salary
Salary by agreement
Pay period
Annual

Company

Industry
AI

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason