You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Cresta

AI Agent Evaluation Engineer

  • Remote
  • 3-6 years
  • English (B2)

Salary

Not stated

Similar roles pay $95K - 160K a year · our estimate

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company's AI Receptionist answers phones for small businesses so calls are not missed. It handles millions of conversations a month across hundreds of customers, and the company needs a full-time owner to judge whether those conversations are actually good.

You will be the company's first AI Agent Evaluation Engineer. You will set the standard for quality across every agent the team runs, build the systems that measure it in every conversation, and be the person people trust when they ask how good the agent really is.

The job mixes hands-on conversation review, careful evaluation design, and engineering work. You will read and label real calls to see how the agent fails, build automated judges, golden sets and regression suites to measure those failures at scale, and check those systems against human judgment. Fixing problems is welcome but secondary; the main output is an accurate, prioritized view of quality.

The company was started at the Stanford AI Lab and has raised over $270 million. Its customers include large airlines, cable providers and hotel chains.

Responsibilities

  • Write and maintain the evaluation rubric across three areas: adherence to design, tool use and guardrails; whether the conversation reached its goal; and the conversation experience, covering latency, turn-taking, tone and repetition.
  • Write the annotation guidelines and failure taxonomy the team uses, run calibration sessions, and measure how well reviewers agree.
  • Rank failure modes by severity and frequency so engineering and product focus on the most important ones.
  • Review production conversations by hand each week, with help from QA partners. Open-code failures, group them, and decide which groups should become automated evaluations.
  • Trace each failure to its likely cause, such as the prompt, a knowledge gap, a tool or flow bug, or speech recognition, and send it to the right owner with enough evidence to act.
  • Report quality by segment, including language, accent, audio quality and intent, not only in aggregate.
  • Build and run the pipeline that scores every production conversation using automated checks, LLM-based judges and human review.
  • Check every automated judge against human labels before relying on it, and treat judge error as the main measurement risk.
  • Design stratified sampling by intent, language, caller type and audio conditions so reported quality changes are real.
  • Build and maintain the golden set and regression suite that every prompt, model, knowledge or flow change must pass before release.
  • Map which behaviors are tested, partly tested or unmeasured, and close the gaps.
  • Design evaluations for areas standard harnesses miss: handoffs to a human, misuse and unwanted callers, and behavior when upstream systems fail.
  • Set the quality and latency thresholds the agent must meet.
  • Report trends to internal stakeholders and customers with uncertainty stated and bad news shown clearly.
  • Work with forward-deployed engineers, backend engineers, product managers and the voice and language team to turn findings into prioritized recommendations.
  • Document methods so results can be reproduced and the practice survives beyond any one person.

Requirements

  • 3+ years of professional experience in software engineering, applied ML or data-intensive research, including at least 2 years working hands-on with LLM-based systems in production.
  • A bachelor's or master's degree in computer science, computational linguistics, cognitive science, statistics or a related field, or equivalent practical experience.
  • You have built and run an end-to-end evaluation system for a production LLM product, covering rubric, human labels, automated judge and regression suite.
  • You understand evaluation methods well: what golden sets, judges, regression suites and calibration are for, and when each one fails. You are not tied to a particular tool, and you will work with the company's in-house tooling.
  • You understand how LLM agents work, including prompts, tool calls, context handling and multi-turn state, and you can read a trace and explain what the agent did and why.
  • You have strong Python skills and experience with LLM APIs, and you can build and ship evaluation tools on your own. Experience with fine-tuning or ML frameworks is not required.
  • You can write a rubric that independent reviewers apply the same way, and you can measure whether they do.
  • You can turn an open-ended quality question into a test, deciding what to measure and how, instead of following a set plan.
  • You are willing to do hands-on work every week, yourself and by directing QA partners: listening to calls, labeling conversations and building a failure taxonomy from scratch.
  • You are professionally fluent in at least one language other than English, enough to evaluate multilingual agent behavior directly.
  • You write clearly and produce guidelines that others follow.
  • The first step after an initial resume review is a self-paced CoderPad technical assessment, followed by conversational interviews.

Conditions

Pay is location-based and reflects the market and each person's experience. The posted base salary range is what the company expects to pay in a given location, and final offers depend on experience, skills, education and geography. Total compensation includes equity and a full benefits package for you and your family.

The company may use AI tools in recruiting, always with human review. Questions or opt-out requests go to the company's site with the subject 'Opt-Out'.

Candidates may not record interviews or use AI transcription by phone, notetaker, AI glasses or any other device. Exceptions can be requested as an accommodation.

Recruiting emails come only from the company's site domain. Ignore outreach claiming to be from the company through any other channel.

Benefits

  • Medical, dental and vision coverage for you and your family
  • Flexible paid time off
  • Paid parental leave for all new parents
  • Retirement savings plan
  • Budget to set up a home office for remote work
  • Monthly wellness and communication stipend
  • In-office meal program and commuter benefits for onsite staff

Where you’d work

Fully remote

You can work from

  • Canada
  • United States

About the company

Cresta

12 of their 12 open roles are remote

Your chances

We've checked whether it's still hiring and how crowded it is.

  • 19 checks run
  • 0 red flags

Still hiring?

13 checks

No red flags

How crowded?

6 checks

No red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details11 facts · Role, Location, Compensation, Employment, Requirements
Tech stack
  • Python
  • LLMs
Type
Full-time
Equity
Equity offered
Region
Canada
Pay period
Annual
Show 6 more factsShow less

Role

Category
Development
Experience
3+ years
Tech stack
  • Python
  • LLMs

Location

Work model
Remote
Region
Canada
Remote from
  • Canada
  • United States

Compensation

Salary
Salary by agreement
Pay period
Annual
Equity
Equity offered

Employment

Type
Full-time

Requirements

Languages
  • English (B2)

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason