Agent Evaluation and Self-Improvement Engineer

Description

Autonomous agents now do real work across NVIDIA, and every piece underneath them keeps changing: models, harnesses, tools, skills, and plugins. Evaluation is how a team knows whether a change made an agent better or quietly made it worse. We're looking for a principal engineer to set evaluation strategy at every layer, from individual skills and MCP servers to whole agents and products, and to design how agents use those results to improve themselves.

Agents can already rewrite the skills and plugins they run on, then test whether the change helped. Making that loop trustworthy is the hard part. What does it take for an agent to get better on its own, and prove the improvement is real? We're building a system that answers that question for every agent at NVIDIA. Come build it with us!

What you'll be doing:

Build evaluation that works the same across harnesses, with automated checks on every change that catch regressions before users do.

Mine production traces for failures, new usage patterns, missing skills and tools, cost and latency hot spots, and successful runs. Turn them into eval cases and improvement signals.

Evaluate the eval datasets themselves: coverage of real usage, duplicate cases, and how closely they match production.

Build the self-improvement loop for evolving harnesses: agents propose changes to their own skills, plugins, and harness configuration, and sandboxed evals test each one. Only measurable gains ship, with human sign-off and limits agents can't change.

Create realistic enterprise environments: sandboxed replicas of issue trackers, code repositories, chat, and internal APIs, filled with synthetic or anonymized data, where agents safely run long, end-to-end tasks.

Measure multi-agent swarms as a team: how agents split work, whether handoffs succeed, and whether permissions hold at every hop. Record how long each task takes and how many tokens it uses.

Keep evals honest: calibrate model and agent judges against human labels. Catch agents that cheat, such as working backward from expected answers or reaching data and tools they shouldn't access. We work closely with product, platform, and hardware teams throughout NVIDIA, and we help shape open evaluation standards and benchmarks across the industry.

What we need to see:

BS or MS in Computer Science, Engineering, or related field (or equivalent experience)

12+ years building large-scale systems, including evaluation systems for ML or agents and the data pipelines that feed them

Hands-on experience building and evaluating agents on an open or closed agent harness

Solid grounding in experiment design: statistical significance, variance across repeated runs, and A/B testing

Experience setting technical direction across teams as a senior individual contributor

Proficiency in Python, Go, Rust, or similar

Ways to stand out from the crowd:

Built self-improving or self-evolving agent systems that get better from their own eval results

Shipped an agent benchmark, eval framework, or simulated test environment adopted by other teams

Detected reward hacking or eval gaming, or built oversight for systems that modify themselves

Built domain evaluations, such as for chip design or hardware verification

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 224,000 USD - 356,500 USD for Level 5, and 272,000 USD - 431,250 USD for Level 6.

Responsibilities

• Applications for this job will be accepted at least until October 20, 2026.
• This posting is for an existing vacancy.
• NVIDIA uses AI tools in its recruiting processes.
• NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.