You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Lila Sciences

ML Research Engineer, Performance Optimization

  • Office
  • 3-6 years

Salary

$148,000 - 240,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The AI Research team at the company is hiring an ML Research Engineer to speed up iteration for mid- and post-training runs on LLMs with 1T+ parameters, running on the latest NVIDIA hardware.

The work covers getting the most out of each GPU during a single training run, and building those optimizations so they stay maintainable, reusable, and easy to adapt, so other researchers can confidently use the newest optimized stack.

The company is building a platform where AI and automation develop together to speed up scientific discovery.

Responsibilities

  • Profile workloads and write kernels, handle long context, and reduce memory footprint to improve GPU compute performance.
  • Improve cluster-level performance by optimizing collective and MoE frameworks, overlapping communication with compute, using RDMA, IB, or EFA, and working on P/D disaggregation.
  • Manage mid-training data at scale across storage, streaming, tokenization, and processing.
  • Speed up the async RL cycle, including sandbox and environment interfaces, long rollouts, large token metadata such as router replay, and weight synchronization.
  • Make sure optimized configurations are tested and accessible, and can be adapted to research tasks so the whole team moves quickly.
  • Work with researchers to map throughput versus accuracy tradeoffs, such as lower-precision compute or higher async staleness, and find the best operating points.

Requirements

  • A BS or MS in Computer Science or a related quantitative field plus 4+ years as a software or machine learning engineer, or a PhD in a relevant research area plus 1+ years of work experience.
  • Experience with LLM inference libraries such as vLLM and training libraries such as Megatron, including usage and throughput tuning.
  • Experience running LLM inference or training at scale on multi-node GPU clusters.
  • Familiarity with core reinforcement learning concepts.
  • A track record of strong software engineering, with clean APIs, well-tested code, and sound abstractions.
  • Hands-on experience supporting and maintaining production libraries that other engineers and researchers depend on.
  • Strong CI/CD skills, including designing integration test suites, build pipelines, and automated quality gates for multi-package systems.
  • Strong interest in agentic systems and awareness of current approaches to LLM tool use and autonomy.
  • Nice to have: experience with kernel DSLs and frameworks such as CUDA, CuTe, CuTile, or Triton.
  • Nice to have: experience with GPU networking frameworks such as NCCL and NIXL, and fabrics such as NVLink, IB, and EFA.
  • Nice to have: experience with distributed systems or compute frameworks such as Ray, Flyte, or Airflow.
  • Nice to have: familiarity with containerized compute backends such as Docker or Modal, and sandbox orchestration.
  • Nice to have: experience with Ray actors, Ray Serve, or similar actor-model frameworks.

Conditions

Base pay is competitive, with bonus potential and early-stage equity. The final offer depends on your background, expertise, and expected impact.

Expected base salary range: $148,000 to $240,000 USD.

USD salary ranges apply only to U.S.-based roles. International salaries follow local market rates.

Equal employment opportunity is offered regardless of protected characteristics.

Benefits

  • Full-time U.S. employees get medical, dental, and vision coverage, employer-paid life and disability insurance, flexible time off with company-wide holidays, paid parental leave, an education assistance program, commuter benefits including bike share for office staff, and a subsidized company lunch program.
  • Full-time employees outside the U.S. get a benefits package tailored to their region.

Where you’d work

From the office

About the company

Lila Sciences

  • Industry: AI

Offices in United States, San Francisco, United States

Also hiring in London, United Kingdom

1 of their 22 open roles is remote

Your chances

We've checked whether it's still hiring and how crowded it is.

  • 21 checks run
  • 0 red flags

Still hiring?

14 checks

No red flags

How crowded?

7 checks

No red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details12 facts · Role, Location, Compensation, Employment, Company
Tech stack
  • Docker
  • Airflow
  • LLMs
  • CUDA
Type
Full-time
Equity
Equity offered
Industry
AI
Specialty
ML
Region
United States
Show 6 more factsShow less

Role

Category
Data & Analytics
Specialty
ML
Experience
4+ years
Tech stack
  • Docker
  • Airflow
  • LLMs
  • CUDA

Location

Work model
Office
Region
United States
Offices
  • United States
  • San Francisco, United States

Compensation

Salary
$148,000 - 240,000 / year
Pay period
Annual
Equity
Equity offered

Employment

Type
Full-time

Company

Industry
AI

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason