AI Quality Engineer

Description

The team is hiring an AI Quality Engineer for the company's services excellence centre in Poland, where the goal is to make AI-powered products more trustworthy in terms of quality, safety and reliability.

The work is hands-on. You will study how AI systems behave, find weak spots before customers do, and build dependable ways to check whether AI solutions deliver what they were designed to do.

You will work on conversational AI, autonomous agents and other AI-powered systems, alongside engineers, product teams and business stakeholders, so that solutions are robust and ready for production.

Seniority levels may vary for this role.

Responsibilities

• Turn business goals, user outcomes and operational risks into measurable evaluation criteria.
• Define what success, partial success and failure look like, including the range between them.
• Design evaluation methods for conversational, multi-turn and non-deterministic AI systems.
• Build and maintain evaluation datasets that cover common, rare, boundary and high-risk cases.
• Check dataset quality for coverage, class balance, duplicates, leakage, label accuracy, differences between synthetic and production data, and freshness.
• Choose suitable manual, deterministic, model-based and statistical evaluation methods.
• Group errors by category, user journey, risk level and likely root cause.
• Tell model failures apart from failures in retrieval, chunking, ranking, prompts, tools, guardrails, integrations, source data and evaluators.
• Validate automated evaluators, including LLM-as-a-Judge, against human judgement.
• Spot blind spots, noise, bias and misleading aggregates in evaluation results.
• Set up baselines, controlled comparisons and regression tests to compare system versions fairly.
• Rank tests by the likelihood and impact of failure, not only by coverage.
• Report results, uncertainty, limitations and residual risks so they support release and product decisions.
• Watch for new failure modes arising from system changes, production data and research.

Requirements

• Professional experience in quality engineering, software testing, test automation, data science, software engineering or a related field.
• Experience with AI-powered solutions, through building, testing, evaluating or supporting them, or a strong interest in doing so.
• Solid Python skills and the ability to write clean, maintainable code for analysis, automation or testing.
• Curiosity about how AI systems behave, how reliable they are and how to measure their quality.
• Strong analytical skills and the ability to investigate problems and find root causes.
• Ability to turn business goals, customer outcomes and risks into practical evaluation approaches.
• Clear communication, including explaining findings to technical and non-technical people.
• A proactive, curious attitude and real interest in AI, new technologies and system robustness.
• Experience evaluating, testing or monitoring AI-powered systems in production.
• Experience working with structured data, especially JSON, logs and tabular evaluation results.
• Understanding of LLM behaviour, tokenisation, context limits, parameters and non-determinism.
• Understanding of multi-turn context, RAG, grounding, faithfulness, hallucinations, guardrails, attacks such as prompt injection, and tool or function calling.
• Good software engineering practice: Git, modular code, configuration management and automated tests.

Conditions

You will be employed under a contract of employment.

The role is hybrid from Poland, with a few days per month in the office in Warsaw's Gdański Business Center, while most colleagues combine flexible remote work with office time.

The company applies equal treatment, is friendly to people with disabilities, and does not ask candidates for date of birth, marital status or a photo on the CV.

Benefits

• Performance bonus.
• Private medical care.
• A cafeteria benefits system, including a multisport card.
• Extra parental privileges: 6 weeks of leave for the second parent and shorter working time after parental leave.
• Employer-funded group insurance.
• Employee share plan.
• An extra day off for your birthday.
• Paid time off to volunteer.
• Additional leave that grows with your years at the company, up to 5 days more per year.
• Wellbeing leave of two weeks of paid leave after 10 years of service.
• Psychological support consultations.
• Access to e-learning platforms.
• An internal employee referral programme.
• Additional entitlements for people with disabilities.