Software Engineers: Paid Interview on AI Evaluation Tasks

Description

The company is running a paid study on how realistic and accurate coding environments are when used to test AI agents. It is building a broad set of programming tasks and evaluation harnesses to measure AI performance more precisely, and your input will influence how these agents are compared against real engineering standards.

The session is remote. You will look over several programming tasks and their evaluation harnesses, judge how technically accurate, complex and realistic each one is, and walk through the logic of the environments to spot possible flaws. You will then say how these environments compare with common industry practice.

The study suits software engineers with hands-on work building, reviewing or testing realistic programming tasks. Full-stack developers, backend engineers, test automation engineers and systems architects who know evaluation harnesses are all welcome. The strongest candidates know what makes a coding challenge robust, verifiable and technically sound.

Responsibilities

• Assess how realistic and high quality a set of programming tasks is
• Review coding environments and test harnesses built for AI evaluation
• Talk through possible weaknesses in the code structures provided
• Give feedback on how hard and how realistic each challenge is

Requirements

• Professional experience as a software engineer or developer
• Hands-on work building or checking programming tasks
• Experience reviewing code and evaluation harnesses
• Able to discuss technical architecture and testing approaches comfortably

Conditions

Pay is $75 per hour

The session is remote