Full-Stack Engineer, Agent Evaluation & Reliability

Description

The company's AI/ML engineering team builds infrastructure that tests agents and helps them improve over time.

This role leans toward backend work but also covers full-stack features from start to finish, with the goal of making agent workflows more dependable for enterprise clients.

Responsibilities

• Create backend services that run agent testing and ongoing improvement processes.
• Model the data behind test datasets, experiment runs, agent traces, knowledge versions, findings, and quality criteria that customers define.
• Build and upkeep enterprise integrations covering authentication, OAuth, webhooks, incremental and backfill syncs, rate-limit handling, retries, and monitoring.
• Take features from design and build through deployment, monitoring, customer feedback, and later revisions.
• Work directly with customers to learn their systems, find out why things fail, and convert their specific needs into reusable product features.
• Carry work across the full stack and help shape architecture and product choices.

Requirements

• 2 to 4 years building and shipping production systems in full-stack or backend roles.
• Production experience with TypeScript and either Node.js or Bun.
• Hands-on experience developing machine learning models, on top of your software engineering background.
• Experience at a fast-moving startup, from early stage through Series D.
• Production experience designing PostgreSQL schemas and working with queues or worker systems.
• A record of driving projects to completion and building backend systems that involve data modeling, integrations, observability, or debugging distributed workflows.
• An understanding of why testing nondeterministic agent systems is different from conventional unit testing.
• Useful: experience with agent infrastructure, LLM evaluation, tracing, or agent memory systems, gained in production or through substantial side projects.

Conditions

Base salary ranges from $120,000 to $180,000 per year.

Visa sponsorship is available, including H-1B transfers, OPT, and O-1.

On-site in San Francisco, California, United States.