You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
ServiceNow

Staff Machine Learning Engineer, Agent Eval Platform

  • Office
  • 6+ years

Salary

$176,100 - 308,200/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company's AI agents plan, call tools, and change real state in enterprise systems for about 5.5 million employees. Building agents is no longer the hard part; improving them is, meaning knowing whether a change helped, why a conversation failed, and which part of the system to fix.

You will build the loop that lets the agents improve from their own use: finding failures in real traffic, identifying which part of the agent harness is at fault, testing each proposed fix on cases the fixer never saw, and converting every fixed failure into a lasting test. The work targets the system around the model (instructions, tool descriptions, routing, knowledge and memory). The model itself is not trained.

This is a mix of ML and systems engineering. You will decide when a signal is reliable enough to act on and build the infrastructure that produces it reproducibly across hundreds of customer organizations.

This is not a pure model-training role and not a pure QA role. The evaluation methods are still an open research question, particularly LLMs judging LLMs, and the agents take real, irreversible actions in shared, stateful enterprise environments.

Responsibilities

  • Build a shared base judge where each evaluation item has its own rubric, stored as configuration beside the dataset, so authors describe what they want instead of copying a prompt for every eval.
  • Split checks by type: deterministic validators for verifiable world state (such as whether a ticket was created with the right item and sent to the right approver), and an LLM judge for subjective points like whether a clarifying question fit, whether policy was followed, and whether the path was efficient.
  • Make scores report their own confidence so uncertain results go to a human rather than becoming trusted signal.
  • Run a continuing calibration loop against human-labeled trajectories with the annotation team, who handle labeling while you own the calibrated judge. Human-to-human agreement caps how good any judge can get, so improving that agreement is part of the role.
  • Guard against shared blind spots, since the user simulator and the judge are both LLMs and may err in the same direction.
  • When simulation and production disagree, work out why, and keep the test suite refreshed with new production failures so it does not overfit.
  • Build the components that let the agent harness improve from its own use without changing the underlying model.
  • Surface where the agent falls short in real use, including failures that look like successes.
  • Generate scenarios and seeded enterprise environments, with expected outcomes and checks, drawn from real failures or ahead of customer demand. Make sure the output is realistic, sufficiently hard, and does not reveal the answer.
  • Determine why the agent failed and whether the fix belongs in instructions, tools, routing, knowledge or memory.
  • Optimize those components against simulation, not live production traffic.
  • Verify each proposed change on held-out cases, detect regressions that an average would conceal, and keep the component that proposes changes separate from the one that judges them.
  • Turn every fixed failure into a permanent test so the suite strengthens as the agent is used.
  • Keep learnings scoped to a single organization where required, and require admin approval for anything specific to an organization.
  • Build infrastructure that runs this loop reproducibly and at scale across hundreds of customer organizations.
  • Hold the agent to end-state outcomes.

Requirements

  • Required: at least 5+ years in applied ML or ML-adjacent engineering, including systems you shipped, operated, and others depended on.
  • Required: experience turning subjective human judgement into a measurement that teams relied on for decisions and that was validated against human labels. This is the core of the job.
  • Required: you have improved a multi-component system through measurement across pipelines, retrieval, prompts, policies or routing, not only by changing model weights.
  • Required: strong applied ML fundamentals and comfort treating LLMs as components to evaluate and prompt rather than models to train.
  • Required: strong Python and production engineering, including services, data pipelines, and designs that make results reproducible through versioned data, deterministic replay and recorded inputs.
  • Required: you think and write clearly about complex problems, and you can persuade engineers that a number means what it claims and be right about it.
  • Required: a high degree of ownership and a bias toward shipping quickly.
  • Required: you are comfortable with ambiguity and can judge when a measurement is good enough to act on.
  • Required: experience in at least 3 of the following: LLM-as-judge or automated evaluation design with calibration against human judgement; human annotation programs covering rubric authoring, label quality and annotator throughput; experimentation rigor with held-out sets, decision rules set in advance, variance and regression detection; failure analysis at scale, classifying, clustering and attributing errors in logs or traces; replay, simulation or sandboxed test infrastructure; synthetic data generation for evaluation with quality controls; prompt and harness optimization handled as versioned, tested and measured engineering; search ranking, recommender or online experimentation evaluation, including golden-set staleness and offline/online divergence; high-volume trace or event data pipelines.
  • Required at Staff level: own the measurement architecture of the self-improving loop end to end, covering judging, deciding where fixes belong, and checking them. Set the measurement standard the organization uses to decide whether a change ships, act as the technical counterpart to the team building the agent, and mentor engineers on evaluation methodology.
  • Bonus: experience evaluating LLM agents with multi-step trajectories, tool calls and actions in stateful environments.
  • Bonus: experience generating complete agent environments (tasks, stateful worlds and verifiers) automatically at scale.
  • Bonus: published or shipped evaluation methodology that others adopted.
  • Bonus: built eval tooling used by teams outside your own, or an eval product customers use.
  • Bonus: worked on agentic systems that take actions rather than only generate text.

Conditions

Base pay for this location is $176,100 - $308,200, plus equity where applicable, variable or incentive pay, and benefits. Pay depends on qualifications, skill level, competencies and work location, so the base shown is a guideline.

Pay is set by the geographic location of the role and may change with work location.

Benefits include health plans, flexible spending accounts, a 401(k) with company match, an employee stock purchase plan, matching donations, a flexible time-away plan, and family leave programs.

Work persona (flexible, remote or required in office) depends on the nature of the work and the assigned work location. The company may use a third-party service to check the distance between your home and the nearest office.

Equal opportunity applies.

Benefits

  • Health plans, including flexible spending accounts.
  • 401(k) plan with company match.
  • Employee stock purchase plan (ESPP).
  • Matching donations.
  • Flexible time-away plan and family leave programs.
  • Equity where applicable, plus variable or incentive pay.

Where you’d work

From the office

The office

About the company

ServiceNow

  • Industry: SaaS

Office in Mountain View, United States

Also hiring in Frankfurt, Germany, Munich, Germany, United States and 13 more places

69 of their 202 open roles are remote

Your chances

Worth a look before you spend an evening tailoring a CV for it.

  • 21 checks run
  • 1 red flag

Still hiring?

14 checks

1 red flag

How crowded?

7 checks

No red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details13 facts · Role, Location, Compensation, Employment, Company
Tech stack
  • Python
  • LLMs
Seniority
Staff
Type
Full-time
Equity
Equity offered
Industry
SaaS
Specialty
ML
Show 7 more factsShow less

Role

Category
Data & Analytics
Specialty
ML
Seniority
Staff
Experience
5+ years
Tech stack
  • Python
  • LLMs

Location

Work model
Office
Region
United States
Office
  • Mountain View, United States

Compensation

Salary
$176,100 - 308,200 / year
Pay period
Annual
Equity
Equity offered

Employment

Type
Full-time

Company

Industry
SaaS

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason