You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Softprobe.ai

Software Engineer — AI Agent QA & Observability

  • Remote

Salary

$30,000 - 50,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company is a US and China developer-tools startup that builds quality infrastructure for AI agents. It focuses on the business layer, meaning whether agents do the right thing for real users, such as detecting customer emotion.

It publishes two open-source products. The first is automated API testing that records real user traffic and replays it as test cases, so teams need no hand-written suites or costly throwaway environments. Teams at Trip.com, Yeahka, United Imaging, Lexin, ZTO and others use it.

The second is an open-source AI evidence lake built on DuckDB and positioned as an alternative to LangSmith and Datadog. It keeps full prompts, responses, tool I/O and HTTP bodies for the long term as open Parquet, about 200x cheaper than index-heavy observability. It accepts OTLP input and exposes DuckDB SQL, and it underpins the company's agent QA product.

Stack: Java, Rust, DuckDB and DuckLake, OpenTelemetry, MongoDB and Redis.

Responsibilities

  • Build and ship features across both open-source products: the traffic capture and replay engine, and the evidence lake's OTLP ingest, DuckLake storage and query layer.
  • Build the business-layer evaluation side of the agent QA product, including evals, the explorer view, diagnosis and team collaboration on top of the evidence lake.
  • Design evals that score agents on business outcomes rather than only token-level metrics, and make the results clear to the teams running those agents.
  • Work with real production agent traffic, including replaying and regressing against recorded sessions.

Requirements

  • Expert in AI agents, which is the core requirement: you have built, shipped and debugged LLM agents in production, not only demos. You understand failure modes such as bad tool calls, lost context, wrong escalation and hallucinated confidence, and how to measure and fix them.
  • Solid backend engineering skills, and you can take a feature from design through to production.
  • Hands-on work with LLM evaluation: you have built evals or benchmarks, or run red-teaming, and used the results to improve a product.
  • Working knowledge of observability and tracing (OpenTelemetry, structured logging, trace analysis) and of data systems for large-scale storage and querying.
  • Open-source contributions, especially in testing, observability or agent frameworks.
  • Experience with record-and-replay testing systems.
  • Proficiency in Rust and/or Java, plus DuckDB and columnar formats such as Parquet.
  • Experience building customer-facing AI for support, sales or service agents, and an understanding of the quality those users expect.

About the company

Softprobe.ai

Your chances

Worth a look before you spend an evening tailoring a CV for it.

  • 20 checks run
  • 2 red flags

Still hiring?

13 checks

1 red flag

How crowded?

7 checks

1 red flag

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details7 facts · Role, Location, Compensation
Tech stack
  • Java
  • SQL
Specialty
Manual
Region
United States
Pay period
Annual
Show 3 more factsShow less

Role

Category
QA & Testing
Specialty
Manual
Tech stack
  • Java
  • SQL

Location

Work model
Remote
Region
United States

Compensation

Salary
$30,000 - 50,000 / year
Pay period
Annual

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason