Software Engineers: Paid Code Review for AI Agent Evaluation

Description

The company is running a paid study on the coding environments and programming tasks used to benchmark AI agents. Evaluation harnesses are built so that models are tested against realistic software engineering scenarios, and the results help agents handle complex coding goals.

The session is remote. You will go through a set of coding tasks and their evaluation harnesses and judge whether the problems match real-world software engineering work. You will check the logic, test cases and overall structure of the environments. Your feedback is collected through a guided conversation and screen sharing.

The study is aimed at practicing software engineers who have built and tested complex systems. Candidates should have direct experience writing test suites, evaluation harnesses or thorough code reviews. Backend engineers, full-stack developers, machine learning engineers and software architects are all welcome.

Responsibilities

• Judge whether programming tasks are realistic and appropriately hard
• Check the logic and test cases inside the evaluation harnesses you are given
• Decide whether a coding environment actually measures software engineering ability
• Talk through your reasoning as you work through complicated code structures

Requirements

• Current professional experience as a software engineer
• Hands-on work building or maintaining test suites and evaluation harnesses
• Able to read code and explain technical ideas out loud
• Familiar with complex real-world software architecture

Conditions

Pay is $65 per hour

The session is remote