You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Prime Intellect

Research Engineer - Pre-training Data

  • Hybrid

Salary

$150,000 - 350,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company is building an open superintelligence stack: infrastructure that frontier AI labs normally build for themselves, offered to AI teams everywhere.

Its platform, called Lab, combines compute, environments, evaluations, secure sandboxes, high-performance training and deployment for post-training at frontier scale, covering SFT, RL, tool use, agent workflows and production models that keep improving.

The company trains open frontier models end to end for long-horizon work such as autonomous research, and uses the same platform internally for that work.

It has raised $150M from Founders Fund, Radical Ventures, NVIDIA and operators from many leading AI, infrastructure and enterprise companies.

The role sits in the data work behind the next generation of the company's open models. You will locate strong sources, turn them into corpora ready for training, and work with researchers to find out which changes actually improve the model. The work covers broad pre-training as well as the data for reasoning, coding and long-context ability.

Responsibilities

  • Locate, assess and gather useful training data from the web, code and technical sources.
  • Build scalable pipelines that take raw data through extraction, cleaning, deduplication and filtering into corpora ready for training.
  • Create checks for data quality, coverage and contamination, and use the results to improve collection and processing.
  • Work with researchers on data mixtures and sampling for pre-training and midtraining.
  • Prepare and assess data for long-context learning, reasoning and coding, and test synthetic data where it proves useful.
  • Run controlled data experiments and use model evaluations to decide which sources, mixtures and processing choices to scale up.
  • Keep large-scale data processing efficient, reproducible and recoverable, with clear source provenance and dataset versioning.

Requirements

  • Strong Python skills and a track record of building reliable data-processing or distributed systems.
  • Experience with large datasets and the practical trade-offs between storage, compute, throughput and reproducibility.
  • Ability to find data-quality problems and turn research goals into measurable checks and experiments.
  • Comfort owning a changing technical problem from start to finish while working closely with researchers to test assumptions and improve the system.
  • Depth in data acquisition, scalable processing or data research is a solid starting point. Not every candidate is expected to have worked across all of these.
  • Relevant experience may include the following:
  • Web crawling, document extraction, code data collection or large-scale corpus curation.
  • Deduplication, text classification, quality scoring, synthetic data or training-data contamination analysis.
  • Data-mixture experiments, curriculum learning or measuring how dataset changes affect model performance.
  • Preparing datasets for language-model pre-training, continued pre-training or midtraining.
  • Working with distributed processing frameworks such as Spark, Ray or similar systems.
  • Work with multilingual, mathematical, scientific or repository-level code data.
  • Familiarity with tokenization, training data loaders and the needs of large model-training runs.

Conditions

Flexible arrangement: you can work remotely or in person at the company's San Francisco office.

Benefits

  • Cash compensation of $150-350k, plus equity incentives tied to the company's growth and your impact.
  • Full medical, dental and vision coverage for you and your dependents.
  • Flexible PTO.
  • A $200/month health and wellness benefit.
  • Visa sponsorship and relocation assistance for international candidates.
  • A professional development budget for courses and conferences.
  • Quarterly team off-sites, hackathons, conferences and learning opportunities.
  • Company-covered meals on days you work from the San Francisco office.

Where you’d work

Part of the week in the office

You can work from

  • United States

Relocation offered

Visa sponsored

About the company

Prime Intellect

  • Industry: AI

Also hiring in San Francisco, United States

2 of their 11 open roles are remote

Your chances

Worth a look before you spend an evening tailoring a CV for it.

  • 23 checks run
  • 6 red flags

Still hiring?

14 checks

2 red flags

How crowded?

9 checks

4 red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details13 facts · Role, Location, Compensation, Employment, Company
Tech stack
  • Python
  • Spark
  • Databricks
  • LangChain
Type
Full-time
Equity
Equity offered
Industry
AI
Specialty
ML
Region
United States
Show 7 more factsShow less

Role

Category
Data & Analytics
Specialty
ML
Tech stack
  • Python
  • Spark
  • Databricks
  • LangChain

Location

Work model
Hybrid
Region
United States
Remote from
  • United States
Relocation
Offered
Visa sponsorship
Sponsored

Compensation

Salary
$150,000 - 350,000 / year
Pay period
Annual
Equity
Equity offered

Employment

Type
Full-time

Company

Industry
AI

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason