You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
ServiceNow

Senior Machine Learning Engineer, Agent Eval Platform

  • Office
  • 6+ years

Salary

$161,300 - 274,200/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company's AI agents take actions, not just produce text. They plan, call tools, and change live state in enterprise systems for 5.5 million employees.

Building agents is no longer the hard part. Knowing whether a change helped, why a conversation failed, and which component to fix is.

The role is applied ML and systems engineering. You build the loop that lets agents improve from real use, without changing the underlying model. Improvements target instructions, tool descriptions, routing, knowledge, and memory.

The methodology for using LLMs to judge LLMs is still an open research question. The work happens on agents that take irreversible actions in shared, stateful enterprise systems.

The team works on the infrastructure that produces trustworthy signals across hundreds of customer organizations.

Responsibilities

  • Design judges and calibrate them. Use one shared base judge with per-item rubrics stored as configuration beside each dataset, so eval authors state intent instead of copying prompts.
  • Use deterministic validators for checkable state, such as whether a ticket was created with the correct item and approver. Use an LLM judge only for subjective points, such as whether a clarifying question was appropriate, whether policy was followed, or whether the path was efficient.
  • Build scoring that reports its own confidence, and route low-confidence judgements to a human reviewer.
  • Run a continuing calibration loop against human-labeled trajectories with the annotation team. They produce the labels, and you own the calibrated judge. Raise the agreement ceiling between human raters, since it caps judge quality.
  • Guard against shared blind spots, since the user simulator and the judge are both LLMs and can err in the same direction.
  • When simulation and production disagree, find out why. Keep the test suite refreshed with new production failures so it does not overfit.
  • Build components that let the harness improve from its own use, without touching the model.
  • Surface where the agent falls short in production, including failures that look like successes.
  • Generate scenarios and seeded enterprise environments, with expected outcomes and checks, from real failures or ahead of customer demand. Confirm they are realistic, hard enough, and do not reveal the answer.
  • Diagnose why an agent failed and decide which harness part should change: instructions, tools, routing, knowledge, or memory.
  • Optimize those components against simulation, not production traffic.
  • Verify each proposed change on cases it was not tuned on. Catch regressions that an average would mask. Keep the proposer separate from the judge.
  • Turn every fixed failure into a permanent test, so the suite strengthens with use.
  • Keep learnings scoped to one organization where required, and require admin approval for org-specific changes.
  • Run the loop reproducibly and at scale across hundreds of customer organizations.
  • Hold the agent to end-state outcomes.

Requirements

  • 2+ years in applied ML, data science, or ML-adjacent engineering, with work that shipped and was used.
  • Experience turning subjective human judgement into a measurement that others, and ideally other models, can act on. This is the core of the job.
  • Solid applied ML fundamentals. You treat LLMs as components to evaluate, prompt, and fine-tune, not to pretrain.
  • Strong Python, with production-grade code rather than notebooks.
  • Clear thinking and communication on complex problems. You will need to persuade engineers that a metric means what you say and that your conclusion is right.
  • High ownership and a bias toward shipping quickly.
  • Comfort with ambiguity, and the judgement to know when a measurement is good enough to act on.
  • Experience in at least 3 of the following areas:
  • LLM-as-judge or automated evaluation design, with calibration against human judgement.
  • Human annotation programs, including rubric writing, label quality, and annotator throughput as a constraint.
  • Search ranking, recommender systems, or online experiment evaluation, including golden-set staleness, offline and online divergence, and rater agreement.
  • Fine-tuning and evaluating small models with SFT, preference tuning, or distillation.
  • Reward modeling, RLHF or RLAIF, or process reward models.
  • Agent trajectory analysis and step-level fault attribution.
  • Prompt engineering run as an engineering discipline, with versioning, tests, and measurement.

Conditions

Base pay for this role in this location is $161,300 to $274,200. Equity may be offered where applicable, along with variable incentive pay.

Actual total pay depends on qualifications, skill level, competencies, and work location, so the base range is a guideline.

Pay is set by the geographic location where the role sits and may change with that location.

Work persona (flexible, remote, or required in office) is assigned based on the nature of the work and the assigned work location.

To check eligibility for a work persona, the company may measure the distance between your primary residence and the nearest office using a third-party service.

Benefits

  • Health plans, including flexible spending accounts.
  • 401(k) plan with company match.
  • Employee stock purchase plan (ESPP).
  • Matching donations.
  • Flexible time away.
  • Family leave programs.

Where you’d work

From the office

About the company

ServiceNow

  • Industry: SaaS

Office in United States

Also hiring in San Diego, United States, Frankfurt, Germany, Munich, Germany and 13 more places

71 of their 204 open roles are remote

Your chances

Worth a look before you spend an evening tailoring a CV for it.

  • 21 checks run
  • 1 red flag

Still hiring?

14 checks

1 red flag

How crowded?

7 checks

No red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details13 facts · Role, Location, Compensation, Employment, Company
Tech stack
  • Python
  • LLMs
Seniority
Senior
Type
Full-time
Equity
Equity offered
Industry
SaaS
Specialty
ML
Show 7 more factsShow less

Role

Category
Data & Analytics
Specialty
ML
Seniority
Senior
Experience
2+ years
Tech stack
  • Python
  • LLMs

Location

Work model
Office
Region
United States
Office
  • United States

Compensation

Salary
$161,300 - 274,200 / year
Pay period
Annual
Equity
Equity offered

Employment

Type
Full-time

Company

Industry
SaaS

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason