You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Robots and Pencils

Principal Applied Scientist

  • Remote
  • 6+ years

Salary

Not stated

Similar roles pay $185K - 260K a year · our estimate

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company needs a Principal Applied Scientist to lead the design and build of an evaluation framework for an enterprise agent that uses a planner. You will work alongside the client's engineers to test the agent across planner models, executor LLMs and inference optimization levers.

Each proposed change, such as a model upgrade, a harness change or a routing strategy, is run as a controlled statistical experiment that produces a measurable dollar outcome.

This is a research-focused, customer-facing role with substantial building. You own both the experimental method and the tools behind it: a scenario-driven user simulator, isolated experiment runs, statistical comparison, automated insights and a locally hosted dashboard the client can take over. The work covers three overlapping streams (evaluation pipeline, grading and statistics, dashboard and experiment campaign) and ends in a recommendation the client can act on.

The aim is to replace guesswork with evidence: which planner, executor and levers to roll out, what each change costs or saves in quality, and what it is worth at production volume. You will also measure how well the planner transfers from data analytics tasks.

Responsibilities

  • Set up the end-to-end evaluation workflow for a planner-executor agent, covering scenarios, user simulation, configuration, execution, grading, comparison and insights.
  • Work with the client's team, who own scenario creation, to define the scenario schema, grading spec and validators so their data fits the framework. Each scenario includes a goal, a persona, a starting state and an expected end state.
  • Build an LLM user simulator with a hidden goal and a stop rule that produces realistic multi-turn conversations, then check its realism against individual production logs.
  • Build isolated, parallel runs using config files, one run per config and scenario, with a harness adapter that sends all model traffic through the client's existing LLM gateway.
  • Package optimization levers as plug-ins and implement a v1 shortlist of 3 to 4, such as cascade-on-failure, cache-aware model switching and prompt caching.
  • Tag each model call at the gateway and send traces to a trace store so tokens, cache usage, latency and cost are attributed to each request.
  • Grade each run against the expected end state as the primary signal, with optional LLM judges for answer and plan quality.
  • Compute verified success, cost per success, p95 latency, and failures traced to the pipeline stage where they occurred.
  • Compare experiments on identical scenarios using paired methods: confidence intervals, per-scenario win, loss and tie counts, and a non-inferiority test that a cost lever does not reduce quality beyond an agreed margin. State clearly when a result is inconclusive.
  • Convert differences into projected dollar savings at production volume using real deployment costs rather than list token prices, and plot the cost-versus-quality frontier.
  • Build an insight engine in which the statistics layer computes every number and an LLM only writes the summary, so each ranked finding links back to its evidence.
  • Build the locally hosted dashboard with Launch, Live, Results, Compare, Insights and Traces views. It must be API-first, deploy with one command, and be ready for later integration into the client's platform.
  • Run the staged campaign: production baseline, model screen, lever screen, then confirmation of the best combination.
  • Analyze generalization by scenario category, optionally alongside a small data-analytics anchor set.
  • Deliver a final recommendation with confidence intervals and dollar impact, plus a runbook and integration guide for go/no-go decisions.
  • Agree scenarios, margins and success criteria with the client's engineers and leadership, and explain findings to both scientists and business stakeholders.
  • Write technical reports and leadership summaries that show business impact.
  • Set experimental standards for the team and mentor engineers on rigorous agent evaluation, including AI-assisted development tools such as Claude Code.

Requirements

  • Hands-on experience with LLM and agent evaluation methods, such as programmatic state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring and user simulation.
  • Experience building or evaluating planner-executor or other multi-step, tool-using agents in production, not only prototypes.
  • Strong background in experimental design and statistical inference, including paired designs, non-inferiority testing, power analysis, and the judgment to call a result inconclusive.
  • Experience with multi-objective optimization or cost-aware ML, including Pareto frontier analysis and converting results into projected dollar savings.
  • Familiarity with LLM routing, model cascades or model selection, and with inference cost levers such as prompt and plan caching, context compaction, reasoning budgets and cache-aware model switching.
  • Experience with observability and tracing for LLM systems, attributing tokens, cache usage, latency and cost per request.
  • Ability to build production-quality research tooling: Python services, containerized and sandboxed execution, and data pipelines.
  • Prior customer-facing or consulting experience scoping and delivering research engagements.
  • Ability to explain technical findings to non-technical audiences, link them to business impact, and go a level deeper when challenged rather than staying abstract.
  • Helpful extra: publications or open-source contributions in LLM evaluation, agents or efficient inference.
  • Helpful extra: experience building internal dashboards or web UIs that a client team can adopt as their own.
  • Helpful extra: hands-on use of an LLM gateway or tracing stack, such as LangFuse or LangSmith, to attribute cost and latency at the request level.
  • Helpful extra: experience in a regulated or high-stakes enterprise domain where evaluation rigor has real consequences.
  • Helpful extra: a point of view on cost-quality tradeoffs gained from running them in production rather than only reading about them.

Conditions

Salary range: $200,00 - $300,000

Where you’d work

Fully remote

You can work from

  • United States

About the company

Robots and Pencils

  • Industry: AI

Also hiring in Mountain View, United States and New York, United States

8 of their 10 open roles are remote

Your chances

We've checked whether it's still hiring and how crowded it is.

  • 19 checks run
  • 0 red flags

Still hiring?

13 checks

No red flags

How crowded?

6 checks

No red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details11 facts · Role, Location, Compensation, Company
Tech stack
  • Prototyping
  • Python
  • LLMs
Seniority
Principal
Industry
AI
Specialty
ML
Region
United States
Pay period
Annual
Show 5 more factsShow less

Role

Category
Data & Analytics
Specialty
ML
Seniority
Principal
Experience
6+ years
Tech stack
  • Prototyping
  • Python
  • LLMs

Location

Work model
Remote
Region
United States
Remote from
  • United States

Compensation

Salary
Salary by agreement
Pay period
Annual

Company

Industry
AI

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason