This vacancy is closed and no longer takes applications. See open Data & Analytics Jobs
Research Engineering Manager, STEM
- Remote
- 6+ years
Salary
Not stated
Description
The company builds large-scale datasets and reinforcement learning environments for frontier AI labs and enterprises. It is headquartered in San Francisco.
It works with frontier labs on high-quality datasets, RL environments and research benchmarks for software engineering, enterprise knowledge work and advanced STEM reasoning. It also deploys agentic AI systems inside critical workflows for Fortune 500 firms in financial services, life sciences, healthcare, retail, automotive and CPG.
The STEM Horizontal team produces the high-signal coding and STEM data that frontier labs train on. It is now building the research engineering behind its next layer: synthetic data that measurably improves models and benchmarks that catch what current evals miss.
The company plans to publish its findings, datasets and benchmarks openly.
Responsibilities
- Manage a group of research engineers and work day to day with research scientists.
- Recruit research engineers, selecting for research taste as well as engineering rigor.
- Stay close enough to the code to review an eval harness line by line or prototype a generator when needed.
- Run the team's operations: project selection, planning, experiment reviews, design docs and clear ownership.
- Coach engineers on experiment design, code quality and presenting results. Give direct, regular feedback and run fair performance cycles.
- Set the team's standards for eval methodology, data quality and reproducibility, so any dataset or result can be regenerated from a commit, a config and a seed.
- Turn hypotheses into experiment plans with scientists, covering what is being tested, the baseline, the ablations and the result that would change the view. Bring your own hypotheses, not only implement theirs.
- Move research prototypes to production scale without losing their scientific intent.
- Own synthetic data pipelines that generate, verify, filter, deduplicate and version data, using programmatic generators, LLM-based generation, automated verifiers and targeted expert review.
- Measure data by its effect on models through ablations and fine-tuning or RL experiments that show which attributes (difficulty, diversity, correctness, coverage) drive gains.
- Build quality controls that hold up at volume: gold sets, LLM-judge calibration against expert labels, sampling audits and agreement metrics.
- Find where frontier models fail and where benchmarks have saturated, then design evaluations for what is missing.
- Own benchmark methodology end to end: task taxonomy, difficulty calibration, graders and verifiers, contamination controls, held-out splits and confidence intervals.
- Ship deterministic, reproducible eval harnesses, including agentic and sandboxed-execution evals, that internal teams and partner labs trust.
- Give the final sign-off on whether a result holds, since partner labs and internal teams act on it.
- Publish benchmarks, datasets and findings with research scientists at venues such as NeurIPS, ICML and ICLR.
- Write code every week on the hardest parts: core verifier logic, eval statistics and the pipeline components others depend on. Expect to spend a third or more of your time on direct technical work.
- Be the person the team turns to for its hardest debugging and method questions, such as why a judge disagrees with experts or whether a gain is real.
- Act as the technical counterpart to researchers at frontier labs: scope requests, set acceptance criteria and explain results and trade-offs plainly.
- Work with strategic project leads, delivery and engineering teams so research reaches client work and the team's platforms.
- Example work: design and ship a benchmark for a capability gap and see partner labs adopt it; build a long-horizon coding pipeline with execution-based verifiers and measure which task properties drive RL gains; calibrate an LLM judge against expert labels and decide where humans stay involved.
Requirements
- 8+ years of hands-on ML or research engineering, at least 5 of them in industry, including 3+ years directly managing research engineers, ML engineers or researchers, covering hiring and performance management.
- Has owned synthetic or curated data pipelines in production that drove training decisions, and can show how quality and downstream impact were measured.
- Has designed and shipped LLM evals or benchmarks that others used for model decisions. Understands contamination, saturation, judge bias, inter-annotator agreement, variance and confidence intervals.
- Plans ablations up front, chooses baselines honestly and can tell a real effect from noise.
- Strong Python. Production experience with PyTorch or JAX. Working knowledge of SFT, DPO, PPO/GRPO and RL with verifiable rewards.
- Has set direction for a team, not only carried out a roadmap handed over.
- Has run distributed data processing and large evaluation or training jobs (for example Ray, Spark or Kubernetes) on AWS or GCP.
- Writes clearly for scientists and non-specialists, and is comfortable presenting results to researchers at partner labs.
- Uses LLM coding tools such as Claude Code, Cursor or Copilot daily, and reviews their output critically. This is a must-have.
- PhD or Master's degree in AI, machine learning, computer science, statistics or a closely related field. PhD preferred. Exceptional equivalent research experience will also be considered, with industry outcomes weighted most heavily.
- Strongly preferred: post-training data, evals or reward modeling experience at a frontier AI lab or a top industry research group.
- Strongly preferred: has built or contributed to a public benchmark or dataset that the community uses.
- Strongly preferred: first-author papers at NeurIPS, ICML, ICLR, ACL, EMNLP or similar venues.
- Strongly preferred: RL environment and verifier design for agentic tasks such as coding agents, tool use, computer use or long-horizon workflows.
- Strongly preferred: data valuation or attribution, measuring what each slice of data contributes to a model.
- Strongly preferred: depth in a STEM domain such as math, physics, chemistry or biology. Not required.
- Strongly preferred: has managed tech leads or managers, or run several concurrent programs.
- Strongly preferred: has directed large expert networks through specs, rubrics and fast feedback loops.
Conditions
Applicants from the European Union should review the company's GDPR notice.
Benefits
- Work on post-training, evaluation and synthetic data alongside leading AI labs and enterprises.
- Build datasets and benchmarks that shape how frontier labs measure progress in coding and STEM reasoning.
- High autonomy, rapid iteration and meaningful commercial impact.
- Work alongside colleagues from Google, Meta, Amazon and other leading technology companies.
- Opportunity to publish research at ICLR, ICML and NeurIPS.
- Startup speed, with ownership and the chance to see your work have real impact.
Where you’d work
Fully remote, from anywhere
You can work from
- Anywhere in the world
About the company
Turing
- Industry: AI
Also hiring in Los Angeles, United States and New York, United States
1 of their 3 open roles is remote
Details11 facts · Role, Location, Compensation, Company- Tech stack
- Python
- PostgreSQL
- AWS
- Kubernetes
- Docker
- Terraform
- PyTorch
- LLMs
- Seniority
- Manager
- Industry
- AI
- Specialty
- ML
- Region
- United States
- Pay period
- Annual
Show 5 more factsShow less
- Python
- PostgreSQL
- AWS
- Kubernetes
- Docker
- Terraform
- PyTorch
- LLMs
Role
- Category
- Data & Analytics
- Specialty
- ML
- Seniority
- Manager
- Experience
- 8+ years
- Tech stack
- Python
- PostgreSQL
- AWS
- Kubernetes
- Docker
- Terraform
- PyTorch
- LLMs
Location
- Work model
- Remote
- Region
- United States
- Remote from
- Anywhere
Compensation
- Salary
- Salary by agreement
- Pay period
- Annual
Company
- Industry
- AI
Something wrong with this vacancy?
Similar vacancies
- Turing
- Turing
- Booz Allen Hamilton
- Early WindowCloses inCompany hidden
- Phantom
- General Motors
- GSK
- 6sense
- Beacon Biosignals
- Reddit
Staff Machine Learning Engineer, Ads Creative Effectiveness
$230,000 - 322,000 / year
- Remote · US
- Senior