Lead Software Test Automation Engineer

Description

We are seeking a Lead Software Test Automation Engineer to define and lead an enterprise-grade test automation strategy across UI, API, data, integration, and platform layers while evolving our quality engineering capabilities for AI and agentic systems. This role will design and scale automation frameworks, establish measurable quality gates for deterministic and non-deterministic product behavior, and drive reliable, maintainable automation that accelerates delivery. The engineer will define validation approaches for probabilistic AI outcomes, including evaluation datasets, threshold- and rubric-based validation, groundedness, regression detection, robustness, performance, and reliability, while embedding automation into CI/CD and production-quality practices.

Essential Duties and responsibilities

Define and own the test automation technical design and risk-based, multi-layer strategy across UI, API, contract, component, integration, end-to-end, data, and platform testing.

Establish automation standards, patterns, and reference implementations covering framework structure, coding practices, reusable components, test data, diagnostics, reporting, and maintainability.

Drive shift-left quality practices through component, contract, API, and pre-merge validation, reducing unnecessary reliance on slow or fragile end-to-end tests.

Design and implement validation approaches for non-deterministic AI and agentic outcomes, including confidence- and threshold-based assertions, semantic and rubric-based evaluation, benchmark datasets, evaluation harnesses, groundedness testing, and regression detection across model, prompt, context, and agent changes.

Define AI quality coverage and measurable acceptance criteria across accuracy, robustness, groundedness, safety where applicable, performance, reliability, and cost-aware execution in partnership with AI Engineering, Product, and Software Engineering.

Build and scale high-throughput automation suites that are reliable, fast, diagnosable, and easy to extend across products and delivery teams.

Improve automation execution time and stability through parallelization, intelligent test selection, dependency control, deterministic test data, and disciplined retry strategies.

Integrate automated validation into CI/CD with appropriate pull-request checks, scheduled suites, deployment gates, and release-readiness criteria.

Standardize test observability, including logs, traces, screenshots or video where appropriate, artifacts, failure diagnostics, and actionable quality dashboards covering execution health, flakiness, coverage, and escaped defects.

Define effective practices for test environments, test data management, service virtualization, external dependency control, and repeatable environment setup.

Serve as the technical authority for software test automation; lead design and code reviews, coach engineers on automation design and code quality, and establish maintainable engineering standards.

Influence product and engineering roadmaps by translating customer, technical, operational, and delivery risks into clear validation priorities and quality investments.

Partner with Product, Software Engineering, AI Engineering, Security, DevOps, and Operations to ensure functional and nonfunctional quality is engineered throughout the product lifecycle.

Use production telemetry, incidents, customer issues, and defect escape patterns to continuously improve automated coverage and prevent recurrence.

Job Qualifications

Education: Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field, or equivalent practical experience. Master’s degree in a relevant technical or business discipline is preferred.

Required Experience / Skills:

8+ years of software quality engineering or test automation experience, including significant ownership of automation frameworks, technical design, and strategy for complex commercial software.

Hands-on experience designing and building scalable automation across API, contract, integration, UI, data, and end-to-end layers.

Strong programming skills in at least one modern language such as Java, Python, TypeScript/JavaScript, or C.

Hands-on experience with modern UI and API automation frameworks such as Playwright, Cypress, Selenium, REST Assured, SuperTest, Postman/Newman, or pytest.

Experience integrating automated testing into CI/CD using platforms such as GitHub Actions, Jenkins, or Azure DevOps, including automated quality gates and release-readiness criteria.

Demonstrated ability to improve test reliability and execution speed through engineering solutions, including reducing flakiness, parallelization, test selection, and effective diagnostics.

Experience defining test data strategies, managing test environments, service virtualization, and dependency control for distributed systems.

Hands-on experience validating AI/ML, generative AI, agentic, or other probabilistic product capabilities in production-oriented software, including evaluation harnesses, benchmark datasets, semantic or rubric-based scoring, groundedness, and regression evaluation.

Practical experience with Agentic Product Development Lifecycle and Spec-Driven Development, including translating specifications and acceptance criteria into traceable automated validation.

Working knowledge of security, performance, accessibility, resilience, and reliability testing within modern software delivery practices.

Knowledge, Skills, and Abilities:

Strong test automation design and systems-thinking skills, with the ability to design automation across application, API, integration, data, platform, and AI/agentic boundaries.

Deep understanding of modern quality engineering practices, including the test pyramid, shift-left validation, contract testing, risk-based testing, and appropriate use of end-to-end automation.

Strong understanding of testing probabilistic AI and agentic systems, including evaluation design, benchmark datasets, thresholds, semantic and rubric-based scoring, groundedness, robustness, regression, and traceability.

Ability to design automation for speed, determinism, maintainability, diagnosability, and reuse while avoiding brittle or over-engineered frameworks.

Strong understanding of CI/CD, automated quality gates, test observability, telemetry, environment strategy, and production feedback loops.

Ability to define meaningful quality measures and dashboards that distinguish product risk and release confidence from simple test counts or pass rates.

Strong coding, code-review, debugging, and root-cause analysis skills with an engineering-first approach to test automation.

Ability to lead through technical influence, establish automation standards, coach engineers, and drive quality ownership across Product, Software Engineering, AI Engineering, DevOps, and Operations.

Strong communication skills with the ability to translate technical and product risk into clear testing priorities, quality decisions, and pragmatic trade-offs.

Ability to balance delivery speed, automation coverage, reliability, maintainability, and business risk without positioning Quality Engineering as a downstream gate.

Our Interview Practices

To maintain a fair and genuine hiring process, we kindly ask that all candidates participate in interviews without the assistance of AI tools or external prompts. Our interview process is designed to assess your individual skills, experiences, and communication style. We value authenticity and want to ensure we’re getting to know you—not a digital assistant. To help maintain this integrity, we ask to remove virtual backgrounds and include in-person interviews in our hiring process. Please note that use of AI-generated responses or third-party support during interviews will be grounds for disqualification from the recruitment process.

Applicants may be required to appear onsite at a the company office as part of the recruitment process.

Conditions

$89,600.00 - $157,000.00 USD

This role is eligible for Bonus.

Compensation range listed is based on primary location of the position. Actual base salary offer is influenced by a wide array of factors including but not limited to skills, experience and actual hiring location. Your recruiter can share more information about the specific offer for the job location during the hiring process.

Additional Information :

The company offers a wide variety of competitive benefits and programs to help meet your needs and balance your work and personal life, including but not limited to: Medical, Dental, & Vision Plans, 401(k), FSA/HSA, Commuter Benefits, Tuition Assistance Plan, Vacation and Sick Time, and Paid Parental Leave. Full details of our benefits are available upon request.