The company's AI agents plan, call tools, and change real state in enterprise systems for about 5.5 million employees. Building agents is no longer the hard part; improving them is, meaning knowing whether a change helped, why a conversation failed, and which part of the system to fix.
You will build the loop that lets the agents improve from their own use: finding failures in real traffic, identifying which part of the agent harness is at fault, testing each proposed fix on cases the fixer never saw, and converting every fixed failure into a lasting test. The work targets the system around the model (instructions, tool descriptions, routing, knowledge and memory). The model itself is not trained.
This is a mix of ML and systems engineering. You will decide when a signal is reliable enough to act on and build the infrastructure that produces it reproducibly across hundreds of customer organizations.
This is not a pure model-training role and not a pure QA role. The evaluation methods are still an open research question, particularly LLMs judging LLMs, and the agents take real, irreversible actions in shared, stateful enterprise environments.
Responsibilities
Build a shared base judge where each evaluation item has its own rubric, stored as configuration beside the dataset, so authors describe what they want instead of copying a prompt for every eval.
Split checks by type: deterministic validators for verifiable world state (such as whether a ticket was created with the right item and sent to the right approver), and an LLM judge for subjective points like whether a clarifying question fit, whether policy was followed, and whether the path was efficient.
Make scores report their own confidence so uncertain results go to a human rather than becoming trusted signal.
Run a continuing calibration loop against human-labeled trajectories with the annotation team, who handle labeling while you own the calibrated judge. Human-to-human agreement caps how good any judge can get, so improving that agreement is part of the role.
Guard against shared blind spots, since the user simulator and the judge are both LLMs and may err in the same direction.
When simulation and production disagree, work out why, and keep the test suite refreshed with new production failures so it does not overfit.
Build the components that let the agent harness improve from its own use without changing the underlying model.
Surface where the agent falls short in real use, including failures that look like successes.
Generate scenarios and seeded enterprise environments, with expected outcomes and checks, drawn from real failures or ahead of customer demand. Make sure the output is realistic, sufficiently hard, and does not reveal the answer.
Determine why the agent failed and whether the fix belongs in instructions, tools, routing, knowledge or memory.
Optimize those components against simulation, not live production traffic.
Verify each proposed change on held-out cases, detect regressions that an average would conceal, and keep the component that proposes changes separate from the one that judges them.
Turn every fixed failure into a permanent test so the suite strengthens as the agent is used.
Keep learnings scoped to a single organization where required, and require admin approval for anything specific to an organization.
Build infrastructure that runs this loop reproducibly and at scale across hundreds of customer organizations.
Hold the agent to end-state outcomes.
Requirements
Required: at least 5+ years in applied ML or ML-adjacent engineering, including systems you shipped, operated, and others depended on.
Required: experience turning subjective human judgement into a measurement that teams relied on for decisions and that was validated against human labels. This is the core of the job.
Required: you have improved a multi-component system through measurement across pipelines, retrieval, prompts, policies or routing, not only by changing model weights.
Required: strong applied ML fundamentals and comfort treating LLMs as components to evaluate and prompt rather than models to train.
Required: strong Python and production engineering, including services, data pipelines, and designs that make results reproducible through versioned data, deterministic replay and recorded inputs.
Required: you think and write clearly about complex problems, and you can persuade engineers that a number means what it claims and be right about it.
Required: a high degree of ownership and a bias toward shipping quickly.
Required: you are comfortable with ambiguity and can judge when a measurement is good enough to act on.
Required: experience in at least 3 of the following: LLM-as-judge or automated evaluation design with calibration against human judgement; human annotation programs covering rubric authoring, label quality and annotator throughput; experimentation rigor with held-out sets, decision rules set in advance, variance and regression detection; failure analysis at scale, classifying, clustering and attributing errors in logs or traces; replay, simulation or sandboxed test infrastructure; synthetic data generation for evaluation with quality controls; prompt and harness optimization handled as versioned, tested and measured engineering; search ranking, recommender or online experimentation evaluation, including golden-set staleness and offline/online divergence; high-volume trace or event data pipelines.
Required at Staff level: own the measurement architecture of the self-improving loop end to end, covering judging, deciding where fixes belong, and checking them. Set the measurement standard the organization uses to decide whether a change ships, act as the technical counterpart to the team building the agent, and mentor engineers on evaluation methodology.
Bonus: experience evaluating LLM agents with multi-step trajectories, tool calls and actions in stateful environments.
Bonus: experience generating complete agent environments (tasks, stateful worlds and verifiers) automatically at scale.
Bonus: published or shipped evaluation methodology that others adopted.
Bonus: built eval tooling used by teams outside your own, or an eval product customers use.
Bonus: worked on agentic systems that take actions rather than only generate text.
Conditions
Base pay for this location is $176,100 - $308,200, plus equity where applicable, variable or incentive pay, and benefits. Pay depends on qualifications, skill level, competencies and work location, so the base shown is a guideline.
Pay is set by the geographic location of the role and may change with work location.
Benefits include health plans, flexible spending accounts, a 401(k) with company match, an employee stock purchase plan, matching donations, a flexible time-away plan, and family leave programs.
Work persona (flexible, remote or required in office) depends on the nature of the work and the assigned work location. The company may use a third-party service to check the distance between your home and the nearest office.
Equal opportunity applies.
Benefits
Health plans, including flexible spending accounts.
401(k) plan with company match.
Employee stock purchase plan (ESPP).
Matching donations.
Flexible time-away plan and family leave programs.
Equity where applicable, plus variable or incentive pay.