The team builds benchmarks and evaluation methods for advanced language models.
Your work helps AI labs and enterprise teams measure what models can do so they can decide how to use them in production.
Responsibilities
Test newly released AI models on a variety of use cases.
Create benchmarks from scratch, covering dataset creation and coordinating labelers.
Design automated methods that score generated text reliably and at scale.
Partner with engineers to build and scale evaluation systems.
Work with AI labs and enterprise teams to turn their evaluation needs into research projects.
Requirements
At least 3 years of full-time AI or ML research, ideally in benchmarking, evaluation, or language models.
A publication record, preferably in NLP or benchmarking, and hands-on work building benchmarks or evaluation methods.
Strong Python skills and the ability to write clean, maintainable code. Experience with Django, Flask, or a similar Python HTTP server framework is a plus.
Experience with PyTorch or TensorFlow, language modeling, LLM infrastructure, and common deep learning techniques.
Solid grounding in computer science and the ability to turn research results into working systems.
Experience working in a team development process using Git, code review, and feedback.
A degree in computer science, machine learning, or a related field. A Master's or PhD is preferred, but candidates with a Bachelor's and 1 to 3 years of applied experience may also be considered.