AI Test Automation Engineer 100% (f/m/d)

Description

You will join the ML & AI ART as a Senior Test Automation Engineer, owning the technical test automation for the AI and MLOps platform that the company relies on.

The platform supports AI features across the company, including retrieval-augmented generation systems, LLM-based assistants, agentic workflows and ML pipelines for business-critical use cases.

You will be responsible for designing, building and running the test automation frameworks. These cover traditional software such as APIs, UIs and data pipelines, as well as the behaviour specific to AI and ML models.

You will also serve as the test automation go-to person within the ART. Testing follows the company's Test Strategy, and you will provide the technical depth and hands-on delivery for your platform.

Responsibilities

• Define and develop the technical test automation approach and framework architecture for the ART, in line with the company's Test Strategy and Test Policy.
• Build reusable, scalable automation patterns such as page objects, API clients and test data builders so engineers across squads can apply them consistently.
• Assess requirements, features and stories for testability during PI Planning, backlog refinement and iteration planning.
• Derive test cases from technical and risk analysis of functional and non-functional requirements, including reliability, performance, security, usability and robustness. Choose test techniques and automation scope based on risk, coverage goals and return on investment.
• Automate test cases with Playwright-Python for UI, requests with pytest for APIs, and Behave or pytest-bdd for BDD and Gherkin. Apply clean code, reuse, readability and stability.
• Design and implement AI and ML test cases, including evaluation pipelines for LLM outputs and test datasets built for AI and ML needs.
• Build and maintain contract tests, for example with Pact, for platform APIs and microservice boundaries.
• Integrate and orchestrate automated tests in GitLab CI/CD pipelines, including merge request pipelines and GitLab runners.
• Use Docker and Kubernetes to set up isolated, reproducible test environments.
• Plan, schedule and trigger test runs across DEV, INT, UAT and pre-PROD, covering regression, smoke, release and on-demand runs tied to merge requests and PI milestones.
• Monitor execution health. Investigate and quarantine flaky tests, and keep false positives low so quality signals stay reliable.
• Triage results, raise defects in Jira with evidence such as logs, traces, screenshots and videos, and report quality status to the squad, Product Owner and Test Manager.
• Produce audit-ready execution evidence, including run metadata, artefacts and reports, following the Test Policy and retention rules.
• Take part in PI Planning, System Demos, Inspect & Adapt and other SAFe ceremonies within the ML & AI ART.
• Keep traceability from Jira features and stories through automated tests and defects to test results, using the company's test management integration.
• Write and maintain Gherkin BDD scenarios linked to story acceptance criteria.
• Work closely with Product Owners, Scrum Masters, ML Engineers, MLOps Engineers and Data Engineers across squads.
• Support audit artefacts, traceability matrices and test progress reporting in line with reporting requirements.
• Support root cause analysis of defects using logs and traces.
• Prepare test data, using synthetic or anonymised data wherever possible.
• Maintain and refactor tests when UI, API, model or prompt behaviour changes, staying within access controls and test environment policies.

Requirements

• Strong hands-on Python test automation with Playwright-Python, Behave or pytest-bdd, and requests with pytest. REST-Assured experience is a practical equivalent.
• Proven ability to design and own test frameworks, including reusable utilities and patterns that keep tests maintainable.
• Hands-on experience running automated tests in CI/CD pipelines, ideally GitLab or another enterprise platform.
• Experience with distributed and parallel test execution, flaky-test management, and reporting tools such as Allure or pytest-html.
• Solid Git and version control workflows, clean code habits and a healthy code review practice.
• Working knowledge of Docker, and basic Kubernetes knowledge of jobs and namespaces.
• Experience testing AI or ML systems is a strong plus. A clear motivation to build this expertise is required, including evaluating LLM outputs, handling non-deterministic responses, and evaluating RAG and agentic workflows.
• Understanding of API design, microservices, event-driven architectures and authentication layers.
• Sound knowledge of SAFe and DevOps principles. Experience working in an Agile Release Train is a plus.
• Experience with Jira for story and feature tracking, and with test management integration through Xray or Agile Hive.
• Ability to connect user journeys, data flows, authentication layers and system boundaries end to end.
• Comfort working within an established Test Strategy and with Test Managers on execution planning, reporting and compliance.
• Awareness of information security, data privacy and compliant test data handling in regulated environments.
• Strong ownership. You take automation problems from analysis through execution to resolution with little supervision, and work well as a team player.
• Strong communication skills for working with engineers, Product Owners, Scrum Masters, Architects and Test Managers.
• Well organised, structured and reliable.
• Bachelor's or Master's degree in Computer Science, Engineering or a related field.
• Minimum 5 to 7 years in test automation, with substantial hands-on Python work including framework design, ownership and large-scale test execution, not only script writing.
• Experience in a regulated environment such as financial services, healthcare or pharma is strongly preferred.
• ISTQB Foundation certification as a baseline.
• SAFe (SP, SSM or equivalent) or DevOps certifications are a plus.
• Fluent English is required. German is a plus.