You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Arize AI

Senior AI Product Manager, Evals & Annotations

  • Remote
  • 6+ years

Salary

$200,000 - 250,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

Evaluations are how AI developers understand, measure, and improve their applications. As AI systems grow more complex and agentic, developers need connected workflows to define quality, evaluate behavior, bring in human feedback, and keep improving.

The role is for a technical product manager who owns Evals & Annotations at the company. You will own the full evaluation lifecycle, from creating and running evals to human review, analysis, and iteration. The goal is to join these capabilities into one strong developer experience and help shape evaluations as agents become more autonomous and managed, including through the integration with Dynatrace.

This suits someone with firm product and UX views who shapes the experience rather than only writing requirements. You will explore workflows on your own, build prototypes, and work with Design and Engineering to turn ideas into polished, technically sound products. You should be equally at home in an interaction design discussion and a technical tradeoff discussion with engineers.

The company was acquired, but it still works like an early-stage startup: fast, resourceful, and hands-on. Product, Engineering, and Design work closely, and the founders stay involved in product thinking and design. You will have real ownership and work directly with the cofounder and CPO, in a setting that values initiative, open debate, and jumping in wherever needed.

Responsibilities

  • Own the Evals & Annotations product area end to end: eval creation and execution, human review, annotation workflows, and the wider evaluation lifecycle.
  • Set the product strategy and roadmap, weighing customer needs, core platform work, UX improvements, and new opportunities in AI and agent development.
  • Build a best-in-class evaluation experience. Form clear views on how workflows should work, prototype solutions yourself, and partner with Design on intuitive, polished results.
  • Shape how evaluations evolve as AI development moves toward autonomous and managed agents, including opportunities across the Dynatrace ecosystem.
  • Work with Engineering on technically complex problems in evaluation execution, backend systems, orchestration, and data infrastructure. Take part in technical discussions, implementation choices, and tradeoffs.
  • Talk with customers, typically several conversations a week, to learn how they evaluate and improve AI systems, find the real problems, and test product direction.
  • Work with the cofounders and cross-functional teams on product direction, design exploration, and priorities, applying your own judgment and turning shared thinking into delivery.
  • Make clear priority calls across competing demands and take products from problem definition through launch, adoption, and iteration.

Requirements

  • Significant product management experience owning technically complex developer-facing products. Exact years are flexible for candidates with exceptional technical depth, product judgment, and ownership.
  • Solid understanding of LLM evaluation and AI development workflows, including automated evals, human feedback, experimentation, and how developers iterate on AI applications and agents.
  • Experience with developer tools, AI infrastructure, ML platforms, or closely related technical products, plus a working grasp of backend systems and software architecture.
  • Real technical depth and hands-on engineering ability. You do not need to ship large features as an individual contributor, but you should be able to:
  • Debug issues on your own and read code when needed.
  • Submit small fixes or pull requests.
  • Build prototypes and technical proofs of concept.
  • Propose technical implementations and weigh their tradeoffs.
  • Hold credible technical conversations with engineers.
  • Strong product and UX instincts. You can define and prototype core experiences on your own, not just write requirements, and work well with designers and design engineers to refine and ship them.
  • Strong ownership and initiative with a bias to action. You handle ambiguity and tradeoffs well and get hands-on without waiting for a fully defined roadmap or direction.
  • A collaborative, low-ego style with founders, engineers, and designers. You bring firm opinions, debate constructively, and do well where responsibilities overlap.
  • Experience working closely with technical customers, with clear communication and the ability to move between strategy and execution.
  • Nice to have:
  • Direct experience with LLM evaluation platforms, annotation systems, human-in-the-loop workflows, or agent orchestration.
  • Strong backend or platform engineering background, especially in distributed systems, workflow orchestration, or data infrastructure.
  • Hands-on programming experience and comfort across backend and frontend technologies.
  • Experience with LLM-as-a-judge, custom evaluators, experiment tracking, datasets, or feedback-driven improvement workflows.
  • Prior engineering experience or substantial hands-on software development work.
  • Early-stage or high-growth startup experience, particularly in founder-led settings.

Conditions

The estimated annual salary is $200,000; $250,000, plus a competitive equity package. Actual pay depends on job-related factors such as transferable experience, skills, and qualifications.

The company is remote-first. It has offices in New York City and the San Francisco Bay Area for people in those cities who want to work in person. Everyone else receives a monthly stipend for co-working spaces.

Benefits

  • Comprehensive benefits including medical, dental, and vision coverage, a 401(k) plan, unlimited paid time off, a generous parental leave plan, and other mental health and wellness support.
  • Monthly stipend for co-working spaces for employees outside New York City and the San Francisco Bay Area.

Where you’d work

Fully remote

You can work from

  • United States

About the company

Arize AI

Also hiring in San Francisco, United States

1 of their 2 open roles is remote

Your chances

We've checked whether it's still hiring and how crowded it is.

  • 21 checks run
  • 0 red flags

Still hiring?

14 checks

No red flags

How crowded?

7 checks

No red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details12 facts · Role, Location, Compensation, Employment
Tech stack
  • LLMs
Seniority
Senior
Type
Full-time
Equity
Equity offered
Specialty
Product Manager
Region
United States
Show 6 more factsShow less

Role

Category
Product & Project
Specialty
Product Manager
Seniority
Senior
Experience
6+ years
Tech stack
  • LLMs

Location

Work model
Remote
Region
United States
Remote from
  • United States

Compensation

Salary
$200,000 - 250,000 / year
Pay period
Annual
Equity
Equity offered

Employment

Type
Full-time

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason