Tech Lead Manager, AI Evaluations

Description

The company builds a Work AI platform that helps employees find information, use an AI assistant and run AI agents, all on one secure, open system.

Its offering includes enterprise search, an AI assistant and AI agents, with over 100 connectors to enterprise SaaS tools, a choice of LLMs, and APIs, so organizations can manage AI without being tied to one vendor or facing long setups.

Its Enterprise Graph and Personal Knowledge Graph link people, content and activity to give each employee personalized, context-aware answers. Its agents work across internal and external data, structured and unstructured, past and live.

The product runs inside tools such as Microsoft Teams, Zoom, ServiceNow, Zendesk and GitHub. The company has more than 1,000 employees in over 25 countries and serves customers in 50+ industries.

You will lead a team of applied AI scientists and engineers who build the evaluation strategy and platform used to ship reliable AI products. Evaluation becomes an ongoing part of how models, harnesses, assistants, agents and other AI products are developed and run.

You will turn new AI and ML research into dependable evaluation systems and work with R&D, product and marketing to act on the results. The hard part is no longer writing code but knowing what good output looks like, using production experience to build rigorous evals, and using those evals to keep improving the product.

Responsibilities

• Set the direction of the AI evaluation engineering roadmap, from defining success and building the platform to feeding results into release and product decisions.
• Build reusable, scalable evaluation infrastructure covering datasets, experiments, graders, trace capture and replay, analysis and dashboards. Keep runs reproducible, versioned and cost-aware, and easy for product and engineering teams to use.
• Create and maintain representative eval sets from expert-written examples, governed production signals, synthetic cases and past failures. Cover common workflows, edge cases, long-tail requests, adversarial inputs and key user and domain groups, and keep held-out acceptance sets free from tuning leakage.
• Design a balanced quality framework covering correctness, groundedness, retrieval and context relevance, completeness, instruction following, citation quality, user preference and task success. For agents, also measure tool selection and arguments, multi-step trajectories, policy compliance, recovery and final task state, not only the final answer.
• Combine deterministic checks, model-based graders including LLM autograders, and human review. Write clear rubrics, check automated judges against human judgment, and measure evaluator reliability and limits so teams can read scores correctly.
• Bring evaluation into development and operations: quick CI regression checks, wider pre-release suites, model and prompt comparisons, and sampled production evaluation. Turn user feedback and production traces into new test cases, and use monitoring, canary releases and experiments to catch regressions and drift.
• Connect evaluation results to latency, reliability and cost so tradeoffs can be measured. Help teams diagnose failures by slice and trace, set priorities, and decide on models, routing, prompts and system changes based on evidence.
• Build and lead a high-performing team of applied AI scientists and engineers, coaching and developing its members. Work closely with product, infrastructure, security and application teams to align on evaluation standards and focus on the highest-impact work.

Requirements

• 10+ years in software engineering, ML infrastructure, data systems or a related technical field, including building and running production systems.
• 5+ years of engineering or applied AI team management, with a record of technical leadership, growing engineers and scientists, and delivering cross-functional work.
• Hands-on experience designing, building or running evaluation systems for LLM and GenAI products, including model-based evaluators or autograders, and a working understanding of how to assess model, retrieval/RAG, assistant or agent behavior.
• Understanding of evaluation methods and their tradeoffs: datasets drawn from expert-curated or live traffic, benchmark design, deterministic metrics, rubric and pairwise scoring, human review, judge calibration and validation, human-model agreement, and statistical analysis.
• Grasp of agent evaluation, including tool calling, multi-turn behavior, trajectory quality, task completion and error recovery. Experience with safety testing or red teaming is a plus.
• Strong coding skills in Python or another production language, with solid fundamentals in testing, APIs, data modeling and reliable distributed systems.
• Experience building scalable data or developer platforms, experiment pipelines or CI/CD integrations. Familiarity with cloud infrastructure, Kubernetes, data stores and observability is beneficial.
• Sound judgment on measurement quality: dataset representativeness, judge calibration, reproducibility, uncertainty, and the danger of optimizing for a narrow metric.
• Knowledge of public benchmarks across domains and capabilities is a plus.
• Product-minded and collaborative: you turn vague quality goals into clear measures, explain findings to technical and non-technical audiences, and shape roadmaps across teams.
• MS or PhD in computer science, AI or a related field.

Conditions

Hybrid role: 4 days a week in the Mountain View or San Francisco office.

Base salary range is $250,000 - $350,000 per year. Pay depends on factors such as location, level, job-related knowledge, skills and experience.

Certain roles may be eligible for variable pay, equity and benefits.

The company values a wide range of backgrounds and does not discriminate by race, ethnicity, gender, sexual orientation, religion, age, disability, or civil or family status.

The hiring process includes a short AI-focused exercise or discussion about how you think about, design and use AI to create impact. You may refer to any tools, platforms or workflows you use today. Prior experience with the company is not required.

Benefits

• Competitive compensation.
• Medical, vision and dental coverage.
• Generous time-off policy.
• Option to contribute to a 401k plan for long-term savings.
• Home office improvement stipend when you join.
• Annual education stipend and annual wellness stipend.
• Regular company events and healthy lunches provided daily.