The company behind PlayStation is seeking an experienced Software Development Engineer in Test (SDET) to help ensure the quality, reliability and resilience of Agentic AI systems and backend services.
The company is a subsidiary of Sony Group Corporation. It delivers hardware and network services to over 100 million people and owns widely known entertainment intellectual properties.
You will design and build automated testing for AI and LLM-driven applications, APIs, microservices and distributed systems.
The ideal candidate has strong quality engineering and test automation skills and has tested Generative AI, LLM-based or Agentic AI systems. You will be working with behavior that may not be deterministic, where standard pass or fail checks alone fall short.
Responsibilities
Build and run quality engineering strategies and test plans covering Agentic AI systems, backend services, APIs and service integrations.
Create and maintain scalable automated test frameworks for API, functional, integration, contract, regression and end-to-end testing.
Design testing methods for non-deterministic AI output and define meaningful evaluation criteria.
Spot quality risks early and apply risk-based, shift-left testing throughout the development lifecycle.
Add automated tests to CI/CD pipelines and set up effective quality gates.
Work with software engineers, product managers and other QE teams to improve testability, observability and overall product quality.
Analyze defects and production issues to find root causes, testing gaps and ways to improve QE continuously.
Set quality metrics and report on test coverage, reliability, defects and release readiness.
Test Agentic AI workflows, including multi-step execution, tool selection and invocation, task completion and failure handling.
Check AI-generated responses for accuracy, relevance, groundedness, consistency and expected behavior.
Test AI guardrails, unexpected inputs and failure cases, including wrong tool use and prompt-based attacks.
Verify how AI agents interact with APIs, tools, data sources and downstream services.
Build reusable automation, test data and evaluation scenarios for repeatable AI and service testing.
Validate API contracts, schemas, authentication, business rules, error handling and service integrations.
Investigate failures across distributed systems and tell apart application, integration, environment, automation and AI-related issues.
Take part in defect triage, root-cause analysis, regression testing and release-readiness work.
Requirements
Bachelor's degree in Computer Science, Engineering, Information Technology or a related field, or equivalent hands-on experience.
4+ years in software development engineering in test, quality engineering, software engineering or a related field.
Hands-on experience writing automated tests in Python, Java, JavaScript, TypeScript or a similar language.
Experience with functional, integration, regression, contract and end-to-end testing.
Experience adding automated tests to CI/CD pipelines.
Strong analytical, debugging and root-cause analysis skills.
Experience working in an Agile environment and collaborating across engineering teams.
Strong grasp of testing methods and quality engineering practice.
Strong API and service testing experience, including REST, JSON, schemas, authentication and error handling.
Experience designing and maintaining automated test frameworks.
Experience with Git and CI/CD workflows.
Ability to build risk-based test strategies and pick the right testing layers.
Experience testing distributed systems and service-to-service integrations.
Hands-on experience testing Generative AI, LLM-powered applications or Agentic AI systems.
Understanding of how to test non-deterministic AI behavior and judge outputs beyond exact-match assertions.
Good to have: experience testing Agentic AI workflows, including planning, multi-step execution, tool calling, memory or state management.
Good to have: experience with AI evaluation methods such as evaluation datasets, semantic evaluation, LLM-as-a-Judge, groundedness, relevance or hallucination detection.
Good to have: experience with AI evaluation or observability tools such as LangSmith, DeepEval, Ragas, Arize Phoenix, Promptfoo or similar.
Good to have: experience testing RAG systems or AI agents that call APIs, databases or external tools.
Good to have: experience with contract testing tools such as Pact and API specifications such as OpenAPI or Swagger.
Good to have: experience with event-driven architectures, Kafka, Docker, Kubernetes or cloud platforms such as AWS.
Good to have: experience with performance, scalability, reliability or resiliency testing.
Good to have: experience setting QE metrics, quality gates or cross-team testing standards.
Conditions
This is a contractor role paid at an hourly rate of $58 to $72 USD.
Base pay is set by market benchmarks and location, and may vary with knowledge, skills, experience and location.
Background checks are run at the offer stage for all new hires, which may include criminal checks for some roles, and personal information will be processed for them.
The company works to create an inclusive environment and welcomes applications from everyone.
Benefits
Eligible for medical, dental and vision benefit offerings.