AI Engineer - Data and Evals

Description

The company builds a Work AI platform that helps employees get more done with AI. It started as enterprise search and now offers AI search, an AI assistant and AI agents on one secure, open platform.

The platform connects to over 100 enterprise SaaS tools, lets customers choose their LLM, and provides APIs, so organizations can manage and tailor AI without being tied to one vendor or waiting through long setup projects.

Its Enterprise Graph and Personal Knowledge Graph link people, content and activity, so agents can act on structured and unstructured data, past and real-time, from inside a company and from the wider world.

The company has over 1,000 employees in more than 25 countries and serves customers in 50+ industries.

The team owns data and evaluation science for the AI products. It builds evalsets, LLM judges and benchmarks, the infrastructure that runs evals at scale, and simulated enterprise environments for testing assistants and agents. Work starts from real production usage and feeds improvements across the stack, including agent harnesses, tool use, skill iteration, model routing and post-training of task-specific models.

You will bring new research into production and work with R&D, product and go-to-market teams to turn eval findings into better products and customer results.

Responsibilities

• Create evalsets and benchmarks from expert-written examples, production data, synthetic data and past failures, including edge cases and key user groups.
• Build and grow the eval platform and agent observability: experiment pipelines, trace capture and replay, and dashboards, so runs are reproducible and affordable.
• Build agentic LLM verifiers that check correctness, groundedness, citation quality and task success, and calibrate them against human labels using clear rubrics.
• Run evals at every stage, from CI and pre-release suites to sampled production traffic, and report results with confidence intervals.
• Build enterprise simulation environments with simulated users and realistic company data to test multi-step agent tasks.
• Diagnose failures by slice and trace, and turn eval results into concrete fixes, balancing quality against latency and cost.

Requirements

• 5+ years of software engineering experience with strong Go and/or Python, plus backend skills in APIs, data modeling, testing and distributed pipelines.
• TypeScript is a plus.
• Experience building datasets and benchmarks for LLM use cases, RAG or agents, including evalset curation and synthetic data generation.
• Understanding of eval methods and their tradeoffs: deterministic metrics, LLM-as-a-judge, pointwise vs pairwise grading, human labeling and judge calibration.
• Experience building eval or experiment infrastructure that runs reliably at scale, including CI/CD integrations and observability tools such as OpenTelemetry.
• Strong analytical skills: you dig into traces and failure cases, write SQL, and use data and metrics to explain why quality changed.
• Nice to have: experience with simulation environments and/or model post-training techniques.

Conditions

This role is hybrid, with 4 days a week in the Mountain View or San Francisco office.

The standard base salary range is $200,000 - $280,000 per year. Final pay depends on location, level, job-related knowledge, skills and experience. Some roles may also be eligible for variable compensation, equity and benefits.

Interviews include a brief AI-focused exercise or discussion about how you think about, design and use AI in your work. You may refer to any tools, platforms or workflows you use today, and prior experience with the company is not required.

The company is committed to a diverse, inclusive workplace and does not discriminate based on gender, ethnicity, sexual orientation, religion, civil or family status, age, disability or race.

Benefits

• Competitive compensation and a comprehensive benefits package.
• Medical, vision and dental coverage.
• Generous time-off policy.
• 401k plan with the option to contribute toward long-term savings.
• A home office improvement stipend when you join.
• Annual education and wellness stipends.
• Regular company events and healthy lunches provided daily.