Product Manager - AI Evaluations Tooling

Description

The company's AI portfolio covers 20+ products and is moving into agentic capabilities.

The Evaluations Tooling team is building one shared platform so every AI team can test, track and improve quality in the same way.

The platform standardizes evaluation, helps teams release AI with more confidence, and turns quality results into evidence that scientists, product teams and legal experts can act on.

This is a platform product manager role with a large effect on how teams work.

The ideal person enjoys working with deeply technical users, makes build-or-buy decisions in a shifting vendor market, and holds a high evaluation bar while sequencing the work realistically.

Responsibilities

• Set the vision and direction for the evaluation platform across five areas: batch testing before release, live quality tracking in production, finding and reusing existing evals, letting subject experts write and approve criteria, and testing agents.
• Talk regularly with applied scientists, application engineers and PMs about how they judge AI quality today, and build the platform so teams can measure their own results without waiting on an applied scientist.
• Drive the build of a strong agent testing system: checking full action trajectories, tool call accuracy, rubrics for intermediate steps, managing long judge runs, and simulating multi-step workflows. Speed up agent releases and make this a differentiator.
• Work with legal experts to define evaluation tasks, describe correct outcomes, identify hard cases, and express their judgment as criteria that can be encoded, scored and reused. Design the authoring and sign-off workflow around how these experts actually work.
• Build team dashboards so each AI product team can see its performance, its standing against its quality target, and what changed since the last release, without needing engineering support.
• Lead build-or-buy reviews for tracing tools and commercial eval products, and repeat them regularly as vendors change.
• Work with engineering and applied science to calibrate LLM-as-judge so it is reliable enough to gate deployments.
• Explain technical trade-offs to audiences ranging from scientists to lawyers to executives.

Requirements

• At least 4 years of product management at a technology company, including at least 2 years on technical, platform, infrastructure or data/ML products.
• Working knowledge of the software lifecycle and agile methods.
• Experience running user research with technical audiences and turning the findings into roadmap choices.
• Clear written and spoken communication, with the ability to explain complex technical ideas to technical and non-technical people.
• Tagged skills include agile, cross-functional collaboration, market research and strategy, product lifecycle, stakeholder management and team leadership.
• Preferred: hands-on familiarity with LLM products and how they are evaluated, including rubrics, LLM-as-judge, and offline vs. online evaluation. Agentic system experience is a plus but not required.
• Preferred: background as a platform, developer-tools or data/ML PM serving technical customers, or a proven record of learning a technical domain fast and earning trust from engineers and scientists.
• Preferred: comfort bringing non-engineering experts into technical work and turning their judgment into something a system can use.
• Preferred: strong analytical habits, including setting metrics, testing assumptions, and changing course when the data says so.

Conditions

Remote or hybrid work.

Expected base salary is between $115,000 and $173,000.

The final offer depends on experience, skill set, qualifications and internal pay equity. Offers at the top of the range are uncommon, to leave room for future salary growth.

Total compensation includes a competitive base salary, an annual performance bonus and long-term incentives.

The company is committed to fair and equitable pay.