AI Engineer, Post-Training & Evaluation

Description

The company is an evaluation platform for conversational AI. It helps enterprise teams simulate, monitor and improve voice and chat agents before those agents reach customers.

The product is built around the judge, so this role is central to the company.

You will own the model work for the judges. Your changes go straight into the product that enterprises use to launch reliable voice agents.

The team is on a small Series A stage, and the role includes founding the engineering organization and setting its standards.

Responsibilities

• Design and ship LLM judges that score conversations, covering rubrics, judge prompts and calibration against human raters.
• Build reliability tracking that shows whether judges stay accurate across releases.
• Use LoRA adapters on open-weight models for post-training where training outperforms prompting, trained on data distilled from the company's scored conversations.
• Handle the non-tuning work elsewhere, including tool use, structured outputs and context engineering.
• Keep testing which change actually moves the metric and put your effort there.

Requirements

• 4+ years of experience as an engineer, with strong systems instincts.
• A record of shipping measurable quality gains by combining rigorous measurement with targeted training rather than prompt tweaks alone.
• Experience building eval infrastructure you can trust.
• Experience running LoRA or SFT experiments end to end, including preparing your own training data instead of relying on data cleaned by other teams.
• High agency. Nobody hands you tickets, so you decide what matters next and drive it.
• Ambitious and kind, and willing to help set the bar and grow the team.
• 3+ years of experience.

Conditions

Only US citizens and visa holders are eligible.