Evaluation Engineer

Description

The team tests how well the company's AI applications perform and how dependable they are.

You will build and run tests, keep evaluation datasets current, look into failures, and produce clear evidence of how the AI handles legal and business work.

Responsibilities

• Write automated evaluation cases and helper scripts with common test frameworks, taking direction from senior engineers.
• Build and update test data, source files and reference examples with legal specialists, respecting confidentiality rules, data-use permissions and version control.
• Score answers on accuracy, completeness, citations, retrieval relevance, instruction following and escalation using agreed criteria, and pass unclear cases to experts.
• Test AI workflows end to end, covering document intake, tool calls, structured output and error handling, working alongside the application QA team.
• Inspect outputs and execution traces, reproduce bugs, record evidence, and determine whether a fault lies in the data, retrieval, model or application.
• Compare results across model, prompt and application versions, reporting changes with their uncertainty and limits rather than headline scores alone.
• Check automated graders against human reviews and investigate any gaps.
• Keep regression suites running, help with CI/CD integration, and turn approved production failures into new test cases.

Requirements

• At least 5 years of software development experience, shown through jobs, placements or substantial projects, in languages such as Python, Java or C.
• A solid grasp of large language models, including what they can and cannot do and their typical failure modes, applied with senior guidance to set evaluation criteria, test cases and pass thresholds.
• Skill using coding agents for development, testing and analysis, including prompting, context handling, validation, security, responsible use of generated code, and knowing when to ask for human review.
• Working knowledge of APIs, Git, automated testing, the software development lifecycle, and CI/CD tools such as Jira and Jenkins.
• Eagerness to keep up with fast-moving AI, assess new models, tools and evaluation methods, and apply what you learn sensibly with guidance.
• A basic understanding of statistics and measurement, including sampling, precision and recall, plus strong analysis, attention to detail, clear issue write-ups, and willingness to seek technical or legal advice when needed.
• Experience evaluating AI applications is a plus but not required; academic, internship or project work in this area will also be considered.

Conditions

For roles assigned or hired in New York or California, the company must legally state an estimated pay range, currently $100,000 to $200,000. That range is specific to those two states, is not adjusted for the local geographic differential, and reflects skills, experience, licensure, certifications and business needs, so actual pay depends on each case.