Sr. Software Development Engineer in Test, AI/ML

Description

The company behind the PlayStation brand is a technology and entertainment business and a subsidiary of a larger parent group. Its hardware and network services reach more than 100 million people, and it owns some of the best-known entertainment properties in the world.

The Information Security team protects people, platforms, and products, and gives the business clear, actionable risk and security insights.

This is a senior SDET role focused on ML and AI quality. You will own quality from start to finish on complex, cross-functional projects, covering automated model evaluation, LLM-assisted test generation, and checks on AI-powered workflows. You will work with ML, engineering, product, and infrastructure teams, set quality strategy, build automation frameworks, and lead embedded QE within projects.

Responsibilities

• Set quality strategy and write test plans and automation coverage for ML-powered services and platform components.
• Lead QE on multi-functional projects, covering risk assessment, dependency management, and release readiness.
• Build and maintain scalable automation for backend services, APIs, and ML inference systems in Python, Java, or both.
• Use LLMs and other AI techniques to create, extend, and upkeep high-value test cases for ML workflows.
• Design scenario suites for AI features that cover adversarial prompts, edge cases, ambiguous inputs, and rarely seen user situations.
• Check ML and LLM outputs automatically, including ranking, score distributions, prompt and response quality, hallucination signals, and probabilistic model behavior.
• Run functional, integration, regression, API, end-to-end, performance, and reliability tests on distributed systems.
• Investigate test failures, service anomalies, model inconsistencies, and AI behavior regressions to find root causes and get them fixed.
• Cut flaky tests and make automation faster and more reliable.
• Work with engineering and ML teams to put automated tests into CI/CD pipelines and release flows.
• Work across teams to set shared quality standards, tooling, and guidelines.

Requirements

• Bachelor's degree in Computer Science or an equivalent field.
• 5+ years as an SDET or QE engineer on backend and distributed systems.
• Hands-on use of LLMs to generate, convert, and rank test cases for AI-powered products.
• Experience with AI evaluation tools, prompt evaluation frameworks, model monitoring, or human-in-the-loop review.
• Solid experience testing REST APIs, microservices, and distributed architectures.
• Skill in Python, Java, JavaScript, or a similar language for automation.
• Practical use of automation frameworks such as pytest, JUnit, Selenium, Playwright, Cypress, or Appium.
• Experience with CI/CD systems and test pipelines such as Jenkins or GitHub Actions.
• Experience with cloud and container technology such as AWS, GCP, Kubernetes, or Docker.
• Working knowledge of databases, monitoring, and observability tools.
• Strong grasp of SDLC, Agile, and release processes.
• Strong problem-solving, debugging, and communication skills.
• Preferred: Experience validating ML outputs with statistical analysis or scenario-based testing.
• Preferred: Familiarity with ML infrastructure, data pipelines, or model-serving platforms such as Seldon, KServe, or Ray Serve.
• Preferred: Prior work in content moderation ML, security, fraud detection, or adversarial ML.
• Preferred: Experience testing high-scale, low-latency online services.
• Preferred: Experience with Databricks or similar ML platform tooling.
• Preferred: Familiarity with Node.js, React, or modern frontend technology.
• Preferred: Experience testing mobile, console, or other non-PC platforms.
• What sets you apart: A strong mix of automation engineering and delivery ownership.
• What sets you apart: Ability to drive quality across complex cross-functional initiatives.
• What sets you apart: Practical understanding of testing non-deterministic AI systems and telling model variance apart from quality regressions.
• What sets you apart: Proven skill in risk management and dependency coordination.
• What sets you apart: Ability to influence engineering teams and promote quality guidelines.
• What sets you apart: A commitment to scalable, reliable, and maintainable automation.

Conditions

Estimated base pay range is $114,900 to $172,300 CAD.

Pay is set from market benchmarking and geographic location, and individual pay depends on factors such as knowledge, skills, experience, and location.

The base pay range may vary in line with the hybrid working policy.

Background checks are run at the offer stage and may include criminal background checks for some roles.

Benefits

• Top-tier benefits package.
• Medical, dental, and vision coverage.
• Matching 401(k) and paid time off.
• Wellness program.
• Employee discounts on the company's products.
• May be eligible for a bonus package.