The company needs a Principal Applied Scientist to lead the design and build of an evaluation framework for an enterprise agent that uses a planner. You will work alongside the client's engineers to test the agent across planner models, executor LLMs and inference optimization levers.
Each proposed change, such as a model upgrade, a harness change or a routing strategy, is run as a controlled statistical experiment that produces a measurable dollar outcome.
This is a research-focused, customer-facing role with substantial building. You own both the experimental method and the tools behind it: a scenario-driven user simulator, isolated experiment runs, statistical comparison, automated insights and a locally hosted dashboard the client can take over. The work covers three overlapping streams (evaluation pipeline, grading and statistics, dashboard and experiment campaign) and ends in a recommendation the client can act on.
The aim is to replace guesswork with evidence: which planner, executor and levers to roll out, what each change costs or saves in quality, and what it is worth at production volume. You will also measure how well the planner transfers from data analytics tasks.
Responsibilities
Set up the end-to-end evaluation workflow for a planner-executor agent, covering scenarios, user simulation, configuration, execution, grading, comparison and insights.
Work with the client's team, who own scenario creation, to define the scenario schema, grading spec and validators so their data fits the framework. Each scenario includes a goal, a persona, a starting state and an expected end state.
Build an LLM user simulator with a hidden goal and a stop rule that produces realistic multi-turn conversations, then check its realism against individual production logs.
Build isolated, parallel runs using config files, one run per config and scenario, with a harness adapter that sends all model traffic through the client's existing LLM gateway.
Package optimization levers as plug-ins and implement a v1 shortlist of 3 to 4, such as cascade-on-failure, cache-aware model switching and prompt caching.
Tag each model call at the gateway and send traces to a trace store so tokens, cache usage, latency and cost are attributed to each request.
Grade each run against the expected end state as the primary signal, with optional LLM judges for answer and plan quality.
Compute verified success, cost per success, p95 latency, and failures traced to the pipeline stage where they occurred.
Compare experiments on identical scenarios using paired methods: confidence intervals, per-scenario win, loss and tie counts, and a non-inferiority test that a cost lever does not reduce quality beyond an agreed margin. State clearly when a result is inconclusive.
Convert differences into projected dollar savings at production volume using real deployment costs rather than list token prices, and plot the cost-versus-quality frontier.
Build an insight engine in which the statistics layer computes every number and an LLM only writes the summary, so each ranked finding links back to its evidence.
Build the locally hosted dashboard with Launch, Live, Results, Compare, Insights and Traces views. It must be API-first, deploy with one command, and be ready for later integration into the client's platform.
Run the staged campaign: production baseline, model screen, lever screen, then confirmation of the best combination.
Analyze generalization by scenario category, optionally alongside a small data-analytics anchor set.
Deliver a final recommendation with confidence intervals and dollar impact, plus a runbook and integration guide for go/no-go decisions.
Agree scenarios, margins and success criteria with the client's engineers and leadership, and explain findings to both scientists and business stakeholders.
Write technical reports and leadership summaries that show business impact.
Set experimental standards for the team and mentor engineers on rigorous agent evaluation, including AI-assisted development tools such as Claude Code.
Requirements
Hands-on experience with LLM and agent evaluation methods, such as programmatic state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring and user simulation.
Experience building or evaluating planner-executor or other multi-step, tool-using agents in production, not only prototypes.
Strong background in experimental design and statistical inference, including paired designs, non-inferiority testing, power analysis, and the judgment to call a result inconclusive.
Experience with multi-objective optimization or cost-aware ML, including Pareto frontier analysis and converting results into projected dollar savings.
Familiarity with LLM routing, model cascades or model selection, and with inference cost levers such as prompt and plan caching, context compaction, reasoning budgets and cache-aware model switching.
Experience with observability and tracing for LLM systems, attributing tokens, cache usage, latency and cost per request.
Ability to build production-quality research tooling: Python services, containerized and sandboxed execution, and data pipelines.
Prior customer-facing or consulting experience scoping and delivering research engagements.
Ability to explain technical findings to non-technical audiences, link them to business impact, and go a level deeper when challenged rather than staying abstract.
Helpful extra: publications or open-source contributions in LLM evaluation, agents or efficient inference.
Helpful extra: experience building internal dashboards or web UIs that a client team can adopt as their own.
Helpful extra: hands-on use of an LLM gateway or tracing stack, such as LangFuse or LangSmith, to attribute cost and latency at the request level.
Helpful extra: experience in a regulated or high-stakes enterprise domain where evaluation rigor has real consequences.
Helpful extra: a point of view on cost-quality tradeoffs gained from running them in production rather than only reading about them.
Conditions
Salary range: $200,00 - $300,000
Where you’d work
Fully remote
You can work from
United States
About the company
R
Robots and Pencils
Industry: AI
Also hiring in Mountain View, United States and New York, United States