Research Scientist

Description

The role centres on designing benchmarks and evaluation methods for advanced language models. It mixes hands-on AI research with building practical evaluation systems that help decide how models are tested before they are deployed.

Responsibilities

• Test newly released AI models across many capabilities and use cases.
• Create benchmarks from scratch, which includes building datasets and working with labelers.
• Turn research findings into publications on evaluation methods.
• Make automated methods for judging generated text more effective.
• Partner with engineering teams to put evaluation approaches into production and scale them.
• Work with AI labs and enterprise customers to learn what they need from evaluation and shape research projects around it.

Requirements

• At least 3 years of full-time experience in AI or machine learning research, ideally in benchmarking, model evaluation, NLP, or language models.
• A publication record, preferably in NLP or benchmarking, and an interest in turning research into applied systems.
• Strong Python skills and experience writing clean, maintainable code, including HTTP services built with Django, Flask, or similar frameworks.
• Hands-on work with PyTorch or TensorFlow, language modeling, and AI/ML infrastructure and techniques.
• Experience designing benchmarks and evaluation methodologies.
• Solid computer science fundamentals and experience working through Git, code reviews, and feedback from people in other functions.
• A degree in computer science, machine learning, or a related field. A master's or PhD is preferred, though a bachelor's degree with 1 to 3 years of applied experience may also be considered.
• Experience at an AI startup, research lab, or early-stage company is helpful but not required.

Conditions

Base salary is $150,000 to $200,000 per year.

Visa sponsorship is available.

The role is on-site in San Francisco, California, United States.