Staff Product Manager - AI Evals

Description

The company is the company Technologies, founded in 2024 and based in Arlington, Virginia. It builds offensive cyber operation tools for the U.S. and its allies, and has raised $168M from investors including Khosla Ventures, Accel, Caffeinated Capital, Friends & Family Capital, Point72 Ventures, General Catalyst, and In-Q-Tel.

The company's goal is to make adversaries think twice before attacking American networks, infrastructure, and intellectual property.

This is a full-time, on-site role. Work authorization: U.S. citizenship required; no active security clearance required.

The team builds AI-enabled technology for mission-critical cyber operations. This role focuses on making agentic AI systems measurable, reliable, and easier to improve over time. You will work with AI and engineering teams to shape evaluation capabilities, from defining agent performance and product quality to turning results into workflows and product decisions.

Cyber experience is a plus but not required. The strongest candidates have built or managed agentic systems, either by building evaluations directly or by building the systems teams use to evaluate and improve agents.

Responsibilities

• Own or support the product roadmap for agentic AI evaluations, turning product-area goals into clear priorities and outcomes.
• Work closely with engineering and AI teams to learn their evaluation needs, technical constraints, and user workflows.
• Define how agent performance and product quality are measured, including task success, reliability, and quality.
• Convert technical evaluation needs into product requirements, workflows, and acceptance criteria.
• Help put evaluation workflows and success metrics in place so teams can consistently assess agent behavior and product changes.
• Use evaluation results and feedback to find gaps, guide product decisions, and improve the agent experience over time.

Requirements

• Strong product experience with AI or agentic systems, including building or managing them through product work.
• Experience directly building evaluations for agents, or building the evaluation product, tooling, or system used to assess them.
• Ability to turn technical evaluation needs into product requirements, priorities, and workflows with engineering and AI teams.
• Ability to define or operationalize meaningful measures of agent performance and product quality.
• Strong communication, collaboration, and product judgment in an evolving, technically complex environment.
• Product-minded and technically curious, with a real interest in how agentic systems work and how to improve them.
• Rigorous about evidence and metrics, while recognizing that evaluation methods must reflect real user and product needs.
• Takes ownership, communicates clearly, and balances learning, execution, and iteration.
• Nice to have: experience in cybersecurity, offensive cyber, government, defense, intelligence, or another mission-driven environment.
• Nice to have: familiarity with evaluation approaches such as benchmarks, human or automated evaluation, regression testing, simulation, or analysis of agent traces.
• Nice to have: experience with products that combine autonomous systems with meaningful human judgment and oversight.

Conditions

Full-time, on-site role.

Due to U.S. government contract and security requirements, this role is limited to U.S. citizens.

Some positions may require eligibility for and maintenance of a U.S. Government security clearance. Any active clearance requirement will be stated in the role description.

Benefits

• Health coverage: medical, dental, and vision plan options, plus life, AD&D, and disability coverage options.
• Paid parental leave for eligible full-time employees: 12 weeks for birthing parents, 4 weeks for non-birthing parents, and 6 weeks for adoptive, foster, or intended parents through surrogacy.
• Paid holidays and flexible PTO.
• 401(k) with pre-tax and Roth options, HSA and FSA options, and a dependent care FSA.
• Benefits vary by location, role, and eligibility. Full plan details are shared during the interview and offer process.