Software Development Engineer in Test - Agentic Automation (Remote, CAN)

Description

The company is a global cybersecurity leader that has been building its platform since 2011. Its large distributed, cloud-based systems process almost 3 trillion events a day, and that volume rises every day.

The Detonations and Threat Analysis Platform Engineering team is hiring a hands-on Software Development Engineer in Test. You will build and maintain the functional, integration and performance automation that keeps the platform reliable, and you will help define how AI is tested.

The team's core testing frameworks include agentic test generation and analysis. It validates its AI agents with eval-driven development (EDD).

If you want to set the standard for testing, evaluating and trusting AI agents in production, this role is built for that.

Responsibilities

• Design, develop and maintain automated test suites and frameworks for distributed, cloud-based systems.
• Maintain and extend the agentic test generation and analysis tooling built into the major testing frameworks.
• Build AI-powered workflows and agent skill files that remove manual test work such as test data setup, failure triage and release validation.
• Maintain eval suites, golden datasets and scoring pipelines for AI agents. Score correctness, tool use, trajectory, faithfulness, safety, latency and cost.
• Enforce CI eval gates so every change to prompts, models, tools or orchestration is measured before release. Feed production failures back into the eval sets.
• Find and reduce unreliable AI output, including hallucinated assertions, flaky or redundant tests, wrong failure triage and judge bias.
• Design, run and analyze performance, load and stress tests to locate bottlenecks and confirm that distributed services scale.
• Validate the integration of platform systems, services, data and applications, plus customer-facing features, from end to end.
• Review product, system and agent designs for testability, evaluability, flaws and possible improvements.
• Share knowledge and help set the team's quality and evaluation standards.
• You will work with Java, Node.js, Python and Go; MongoDB, MinIO, RabbitMQ and Kafka; AWS (EC2, S3, DynamoDB), Docker, Nomad, Terraform and Unix/Linux; Git, Jenkins, GitLab CI/CD, Jira and Bitbucket; Locust and k6 for performance testing; and LLM APIs, agent frameworks and LLM eval tooling.

Requirements

• A degree in Computer Science or equivalent experience, with knowledge of OOP, data structures, algorithms and distributed systems.
• At least 5 years in software quality, including at least 3 years of hands-on SDET or test automation development.
• Strong programming skills in at least one of Java, Node.js or Python. Python is preferred for eval and agent work.
• Proven experience building, maintaining and running test frameworks inside CI/CD pipelines such as Jenkins or GitLab CI/CD.
• Experience testing web services and APIs that use REST, gRPC or JSON.
• Hands-on performance and load testing with Locust or k6. You can design realistic workloads and interpret latency percentiles, throughput and error rates.
• A skeptical, verification-first attitude toward AI output, and a clear sense of when to use deterministic assertions versus probabilistic evaluation.
• Proven experience using AI technologies to improve decisions, streamline workflows and processes, raise efficiency and drive business results.
• Bonus: end-to-end eval-driven development experience, including release gating, trajectory and tool-use evals, and calibrating LLM-as-judge against human review.
• Bonus: prompt engineering and context or tool design for agents.
• Bonus: integrating performance tests into CI/CD and using observability tools such as Grafana, Prometheus or CloudWatch.
• Bonus: background in cybersecurity, threat intelligence, malware analysis or sandboxing and detonation environments.

Conditions

This is a remote role based in Canada, and the employer is the company Canada ULC. You must have the legal right to work in Canada.

The base salary range is $120,000 to $170,000 CAD per year. It is in addition to variable or incentive pay, equity and benefits. Pay depends on factors such as relevant experience, skills, certifications and location.

The company is committed to equal pay for work of equal value in its compensation practices.

Selected candidates must pass background checks, which may include criminal record, credit and reference checks.

The company is an equal opportunity employer that welcomes all backgrounds and supports veterans and people with disabilities.

Benefits

• Leading compensation and equity awards.
• Comprehensive physical and mental wellness programs.
• Competitive vacation and holiday time.
• Paid parental and adoption leave.
• Professional development available to all employees, whatever their level or role.
• Employee networks, neighborhood groups and volunteer opportunities for building connections.
• Office spaces with a lively culture and world-class amenities.
• Certified as a Great Place to Work across the globe.