Senior AI Systems Quality Engineer

Description

The company builds an AI and data platform for health plans. It joins fragmented data into one trusted foundation so plan staff can make faster, better-informed decisions on care and cost.

The platform supplies clean, connected, reliable healthcare data for GenAI use cases such as automation, prioritization, and decision workflows. The company has $100M in backing from top investors.

The team values curiosity, collaboration, and boldness. It uses AI and automation thoughtfully and keeps human judgment and clients at the center of decisions.

The role sits in a platform team that makes AI systems in healthcare production-ready, operable, and trustworthy. The work is hands-on engineering, not a manual review step at the end of a release.

Responsibilities

• Build and ship automated validation frameworks, test harnesses, and evaluation pipelines that cover the full AI lifecycle from design to deployment.
• Design and evolve an AI testing platform on Databricks and MLflow that supports repeatable tests, traceability, and audit trails.
• Write large scenario-based test suites, with hundreds to thousands of cases, covering edge cases, long-tail situations, and failure modes across agentic workflows.
• Check orchestration behavior, including tool use, memory, and decision logic, and stress-test non-deterministic behavior before release.
• Set system contracts, guardrails, and safe-degradation patterns at key boundaries so quality is built in from the start.
• Define measurable quality signals for LLM systems, such as grounding, hallucinations, relevance, latency, and cost, and enforce them as automated CI/CD gates.
• Run AI validation automatically on every model, prompt, and code change.
• Build reusable libraries so other teams can adopt the same AI quality practices quickly.
• Own parts of release readiness, including go/no-go criteria tied to measurable quality thresholds.
• Work with AI, platform, security, and delivery teams to turn business needs into clear quality criteria, trade-offs, and confidence levels.

Requirements

• 7+ years of software engineering experience, mostly in backend or platform systems.
• Proven record of building and running AI testing automation in production, not only running tests.
• Experience creating custom validation, evaluation, or testing frameworks for complex distributed systems.
• Strong Python and/or TypeScript skills in modern AI engineering stacks.
• Hands-on work with LLM-based or agentic systems and their non-deterministic behavior.
• Experience with AI testing at scale, including regression frameworks, long-tail evaluation, and broad test coverage.
• Deep knowledge of CI/CD, including embedding automated tests and quality gates in deployment pipelines.
• Solid understanding of AWS cloud-native architecture.
• A track record of treating quality, reliability, governance, and safety as core design principles.
• Working knowledge of security, privacy, and operational risk in regulated or mission-critical settings, including failure modes and recovery.
• Experience with AI testing methods such as evaluating non-deterministic output, drift detection, bias and fairness testing, and regression strategy.
• Proven ability to set measurable trust thresholds for AI systems and enforce them as release criteria, covering query accuracy, hallucination limits, explainability, and PHI-safe behavior.
• Experience working with domain experts to define correct behavior and real-world test scenarios that reflect actual production use.
• Not required: experience with Databricks-native environments and Medallion architecture.
• Not required: experience using MLflow for model evaluation, lineage tracking, and auditability.
• Not required: exposure to observability tools such as Datadog, Prometheus, or Grafana for distributed AI workflows.
• Not required: familiarity with LLM evaluation techniques, guardrails, and policy enforcement frameworks.
• Not required: experience measuring performance, latency, or cost regressions in AI systems.
• Not required: ability to document system behavior and quality trade-offs for technical and business readers.
• Not required: formal certification in AI or ML systems, such as ISTQB AI Testing, AWS ML Specialty, or Google ML Engineer.
• Not required: experience managing prompts, agent behaviors, and orchestration logic as versioned, testable artifacts so they can change quickly without heavy code changes.
• Not required: experience using AI systems to generate and widen test scenarios, including large, varied, and adversarial test datasets.

Conditions

Pay is set by experience, skills, and location. It includes a base salary and eligibility for performance bonuses and equity grants.

The role is remote, with the option to work from anywhere.

Interviews are held by live video with your camera on, in a well-lit, private space without virtual or blurred backgrounds or headsets, unless an approved accommodation applies.

Candidates who advance may be asked to verify their identity, work history, and location. Knowingly false or materially misrepresented information may lead to disqualification or a rescinded offer.

AI-powered tools support parts of recruiting, such as ranking applications by job-related criteria, but recruiters and hiring teams review all applications and make every hiring decision.

The company is an equal opportunity employer. If you need an accommodation during interviews, tell your recruiter in advance.

Benefits

• Unlimited paid time off.
• Comprehensive health coverage with several plan options.
• Equity for every employee.
• A growth-focused environment where your development is a priority.
• A one-time home office setup allowance.
• A monthly cell phone allowance.