Lead Product Data Scientist, Agents

Description

The product data science team shapes how the company measures and improves product and business performance. It works closely with Product, Engineering, Finance and Go-to-Market teams to turn complex data into insights that guide product strategy, customer experience and long-term growth.

The company is expanding its AI, and intelligent agents are becoming central to how businesses reach their customers. One agent helps marketers analyze, build and run their marketing strategies. Another lets brands answer shopper questions across chat, email, SMS and WhatsApp.

These products change quickly, which raises new questions about agent performance, customer value and how AI experiences connect to business results.

The team works AI-first, using emerging technologies and agentic tools to speed up analysis so people can focus on the hardest questions.

This is a hands-on technical leadership role. You will set the measurement and data science strategy for both agents, combining technical depth with product and business judgment. You will define how adoption, engagement, quality, cost and business impact are measured and build the foundations that show what creates customer value. Beyond dashboards, you will tackle new measurement problems, judge whether AI experiences improve customer outcomes, and use experimentation, causal inference and advanced analytics where frameworks are still forming. You will also work with Product, Engineering, Finance and executives on strategy, pricing and monetization, mentor data scientists, and set shared metrics and data standards.

Responsibilities

• Own measurement for both agents from start to finish, building frameworks that link adoption, engagement, agent performance, customer value and revenue.
• Work with Product and Engineering to define what success looks like: effectiveness, task completion, customer outcomes, failure rates, and the trade-offs between quality, cost and business value.
• Create scalable evaluation frameworks using conversation data, agent traces, tool-call results, human review and LLM-based classification to judge quality, spot failure patterns and show how model and agent changes affect results.
• Measure causal impact and incremental value with experiments and advanced methods such as difference-in-differences, matched comparisons, staged rollouts and synthetic controls, to see how agents affect behavior, retention and revenue.
• With Finance and Product, assess usage-based pricing, packaging, credit economics, model costs, conversion and the financial effects of product investment.
• Set trusted metrics and analytical standards: consistent methods, governed data models and evaluation practices that hold up as AI products and their underlying technology change.
• Lead with AI-first analytics, using coding agents, LLMs and new tools for SQL, data modeling, dashboards and conversation classification, while validating their output carefully.
• Build self-service analytics so product managers and other stakeholders can explore agent performance and customer outcomes on their own.
• Work with Engineering and Analytics Engineering on instrumentation, pipelines and dbt models that capture agent interactions, support evaluation and keep data quality steady as new capabilities ship.
• Set the technical direction for agent analytics: review approaches, mentor data scientists, define best practices and raise the bar on rigor and AI-native ways of working.
• Turn findings into clear recommendations that shape product and executive decisions, align stakeholders on metrics, and state uncertainty and trade-offs plainly.
• Keep evaluating new methods, challenge assumptions, and focus effort on the questions that matter most as the agents and AI strategy develop.

Requirements

• You have 10+ years in data science, product analytics, applied statistics or a related quantitative field, with recent hands-on work measuring AI, machine learning or LLM products in production.
• You know product measurement, experimentation, statistical modeling and causal inference well enough to choose and defend a sound method when randomized tests are not possible.
• You have evaluated AI systems in production: designed evaluation frameworks, analyzed agent conversations, used human labeling or LLM-as-judge, and understood how changes to models, prompts and tools affect outcomes.
• You are highly proficient in SQL and in Python or R, can build production-quality analysis, and know dbt and modern analytics engineering.
• You work AI-first, using coding agents and LLMs to speed up analysis while understanding their limits and checking their output for accuracy and reliability.
• You have led technical work by setting direction on complex analysis, mentoring data scientists, reviewing methods and shaping standards across teams.
• You pair statistical rigor with product and business sense, seeing how engagement, AI performance, usage-based pricing and unit economics affect customer and business outcomes.
• You are comfortable with ambiguous problems where metrics or methods may not exist yet, and you build practical frameworks that balance rigor with business needs.
• You have influenced senior stakeholders, aligned cross-functional teams on shared definitions and priorities, and turned complex findings into clear recommendations for technical and non-technical audiences.
• You explain sophisticated concepts well, challenge assumptions constructively, and communicate uncertainty and trade-offs with confidence.
• You do well in fast-moving product environments, adjust as priorities change, and balance analytical depth with the speed of building and iterating on AI products.
• You hold a degree in Statistics, Mathematics, Computer Science, Economics, Finance, Engineering or another quantitative field, or an MBA with a quantitative focus. Equivalent hands-on applied data science experience is equally valued.
• You have measured and improved AI agents, conversational AI or LLM products at a high-growth B2B SaaS or product-focused technology company.
• You have used agent observability and evaluation platforms such as Braintrust, LangSmith or Arize, working with agent traces, tool-call data and evaluation pipelines.
• You have worked with usage-based or consumption pricing, credit models, AI infrastructure costs or LLM unit economics.
• You have built governed metrics, semantic layers, metric trees or standard measurement frameworks used by several product teams.
• You have applied classification, clustering or other advanced methods to large volumes of unstructured text, customer conversations or agent interactions.
• You have built internal AI agents or AI-powered analytics tools that improved team productivity and decisions.
• You know marketing attribution, incremental measurement or the impact of AI-driven customer engagement.
• You have a background in marketing technology, ecommerce, CRM, customer-service software or other data-driven SaaS.

Conditions

The role may require up to 10% travel for purposes such as new hire onboarding, client or partner work where relevant, team meetings and industry events, and travel is coordinated in advance.

The US base pay range is $172,000 - $258,000 USD, the minimum and maximum target salary across all US locations, reflecting labor costs in US markets and set by skills, experience, education or training and work location.

Responsible use of AI, including privacy, security, bias awareness and human-in-the-loop review, is part of AI fluency here.

Accommodations are provided as needed.

The company is an equal opportunity employer that does not discriminate on citizenship or any other protected characteristic, and in Massachusetts a lie detector test cannot be required as a condition of employment.

Benefits

• Total compensation may include the annual cash bonus plan, equity, sign-on payments, and variable compensation (OTE) for sales and customer success roles.
• Health, welfare and wellbeing benefits are offered based on eligibility.