Senior Product Software Engineer - AI Quality Engineering

Description

The company builds generative AI features for professional tax and accounting work, and this role is about making sure those features are accurate, safe and ready for production.

You will design and build the systems that measure retrieval accuracy, citation correctness, answer quality, safety, speed and how well outputs match what customers want. You will work hands-on with test automation, AI evaluation, APIs, data pipelines, observability and release readiness, and you will work alongside software engineers, product managers, UX and stakeholders.

The work goes beyond checking that features behave as designed. Because AI outputs vary, you will decide how quality is measured, build automated evaluation and regression coverage, add quality gates to delivery pipelines, make release-readiness calls, mentor team members and set the quality standards customers rely on.

Responsibilities

• Own the quality engineering approach for large-scale generative AI applications, including RAG pipelines, LLM orchestration, agentic workflows, APIs, ingestion pipelines and AI-driven user experiences.
• Build evaluation harnesses that measure retrieval accuracy, citation correctness, answer relevance, groundedness, hallucination rate, safety, latency, cost and end-to-end behavior.
• Create automated regression suites that catch quality drift caused by changes to prompts, embeddings, chunking, model versions, retrieval settings, orchestration logic or application workflows.
• Work with developers to check AI architecture, service integrations, API behavior, data ingestion, observability, resilience, security and production readiness.
• Lead root-cause analysis of AI quality failures using logs, traces, evaluation results, feedback signals, customer scenarios and domain acceptance criteria.
• Set and maintain measurable quality gates for AI releases, and advise product and engineering leaders on risk, readiness and trade-offs between accuracy, reliability, latency, cost and user trust.
• Write Python test automation for APIs, services, chat workflows, data pipelines and AI evaluation, and run it in CI/CD pipelines.
• Build and maintain observability and dashboards that show model behavior, retrieval performance, operational health, quality trends and customer-facing failures.
• Support responsible AI work, including bias and fairness checks, content safety validation, prompt injection testing, data privacy validation and collecting compliance evidence.
• Turn customer workflows and business requirements from product managers, UX, subject matter experts and stakeholders into testable AI quality criteria.
• Mentor engineers and quality team members on AI evaluation, testing non-deterministic systems, automation design and production-grade engineering practice.
• Keep up with developments in AI quality, LLM evaluation, RAG assessment and LLMOps through research, experiments and community activity.

Requirements

• You have 5+ years in quality engineering, SDET, software engineering, test automation, reliability engineering or a similar role.
• You have 2+ years testing, evaluating or validating AI/ML, LLM, RAG or other non-deterministic systems in production or production-like settings.
• You are strong in Python test frameworks such as Pytest, unittest or similar, and have built reusable harnesses and regression suites.
• You have validated REST APIs, microservices, service integrations, data ingestion pipelines and end-to-end web or chat workflows.
• You understand RAG systems, vector search, embeddings, prompt behavior, model evaluation, groundedness, citation quality and AI failure modes.
• You have defined quality metrics and success criteria for outputs that are probabilistic rather than fixed.
• You know CI/CD pipelines, Git workflows, automated regression testing, release gates and modern engineering practice.
• You have used observability, logging, tracing, dashboards and metrics to monitor production and diagnose failures.
• You can handle ambiguity, break complex AI behavior into testable risks, and explain findings clearly to technical and non-technical audiences.
• You work well in Agile teams and can shape quality practice across product, design, development and domain experts.
• You are committed to responsible AI quality, including trust, transparency, security, privacy, fairness and customer impact.
• Nice to have: experience with LLM observability, tracing and evaluation platforms such as LangSmith, Langfuse, TruLens, Arize/Phoenix, or custom tooling.
• Nice to have: experience testing RAG pipelines on vector databases or search platforms such as Azure AI Search, Pinecone, Weaviate, Elasticsearch or similar.
• Nice to have: experience with LLM application frameworks such as LangChain, LlamaIndex, Semantic Kernel or custom orchestration.
• Nice to have: experience with Azure OpenAI Service, Azure AI Studio, Azure cloud or enterprise AI deployments.
• Nice to have: experience with containers and cloud-native delivery using Docker, Kubernetes and related tools.
• Nice to have: experience with performance, load or resiliency testing tools such as Locust, k6, JMeter or equivalent.
• Nice to have: experience with observability tools such as Azure Application Insights, Grafana, Prometheus, Datadog, OpenTelemetry or equivalent.
• Nice to have: knowledge of LLMOps, model monitoring, A/B testing, human-in-the-loop evaluation or production feedback loops.
• Nice to have: experience validating AI applications that handle sensitive, regulated or professional-domain data.
• Nice to have: background in tax, accounting, legal, financial or other domains where answer accuracy and explainability matter.
• Interviews must be done without AI tools or outside prompts, and virtual backgrounds must be removed. Use of AI-generated answers or third-party help during interviews is grounds for disqualification.
• You may be asked to attend an onsite interview at a the company office.

Conditions

Base pay ranges from $83,400.00 to $145,650.00 USD, based on the primary location of the role.

This role is eligible for a bonus.

The actual offer depends on skills, experience and the hiring location. A recruiter can share the specific figure for the location during the process.

Equal opportunity applies to all candidates.

Benefits

• Medical, dental and vision plans, 401(k), FSA and HSA, commuter benefits, tuition assistance, vacation and sick time, and paid parental leave.
• Full benefit details are available on request.