You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Thinking Machines Lab

Software Engineer, Training Infrastructure

  • Office
  • 3-6 years

Salary

$300,000 - 400,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company builds AI meant to extend human intent and judgment. It trains large frontier models, offers a fine-tuning API that lets people shape models to their own needs, and designs interfaces for better communication between people and AI.

This hire will create the infrastructure that trains the company's frontier models and serves the fine-tuning API.

You will work on the distributed training stack, which runs supervised and reinforcement learning jobs reliably on large GPU clusters. The work covers scheduling, fault tolerance, throughput, and cost.

The API gives users a small set of low-level building blocks so they can write their own training loops, while the team runs the infrastructure underneath. Your task is to keep that model sound as the company adds architectures, users, and heavier workloads such as long-horizon reinforcement learning. The platform must stay fast, stable, and easy to reason about.

Responsibilities

  • Design, build, and run the distributed training infrastructure behind the fine-tuning API and internal frontier training, including scheduling, orchestration, checkpointing, and recovery.
  • Raise training throughput and lower cost through profiling, parallelism, memory work, and tuning at the kernel and communication levels.
  • Get training support running for new open-weights and in-house architectures, including mixture-of-experts, and ship those models to users quickly.
  • Make the platform reliable: detect failures automatically, recover jobs, isolate tenants, and provide observability so long runs survive hardware and network faults.
  • Support multi-tenant LoRA fine-tuning and closely coupled sampling and training for reinforcement learning.
  • Work with the API, framework, and product teams so the infrastructure and the user-facing SDK stay consistent.

Requirements

  • Minimum: 4+ years of software engineering experience, including building and running production distributed systems.
  • Minimum: strong Python skills plus experience in a systems language such as C++, Rust, or Go.
  • Minimum: hands-on work with large-scale model training or inference infrastructure in frameworks such as PyTorch.
  • Minimum: a record of debugging complex performance and reliability problems in distributed environments.
  • Preferred: experience training or serving large language models at scale, including mixture-of-experts architectures.
  • Preferred: experience with LoRA and other parameter-efficient fine-tuning, or with supervised and reinforcement learning pipelines for LLMs.
  • Preferred: experience with cluster orchestration and scheduling, such as Kubernetes or Slurm, and with fault-tolerant checkpointing.
  • Preferred: experience building observability and profiling tooling.
  • Preferred: contributions to open-source ML systems projects such as PyTorch, Megatron, vLLM, SGLang, or similar.

Conditions

Location: based in San Francisco, California or New York, New York.

Salary: $300,000 - $400,000 USD per year, depending on background, skills, and experience.

Visa sponsorship: the company sponsors visas. It cannot guarantee success for every candidate or role, but for the right fit it is committed to working through the visa process together.

Benefits

  • Generous health, dental, and vision benefits.
  • Unlimited PTO.
  • Paid parental leave.
  • Relocation support as needed.

Where you’d work

From the office

Relocation offered

Visa sponsored

About the company

Thinking Machines Lab

  • Industry: AI

Offices in San Francisco, United States, New York, United States

Your chances

Worth a look before you spend an evening tailoring a CV for it.

  • 23 checks run
  • 3 red flags

Still hiring?

14 checks

No red flags

How crowded?

9 checks

3 red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details13 facts · Role, Location, Compensation, Employment, Company
Tech stack
  • Python
  • Go
  • Kubernetes
  • PyTorch
  • C++
  • Rust
  • LLMs
Type
Full-time
Industry
AI
Specialty
Backend
Region
United States
Pay period
Annual
Show 7 more factsShow less

Role

Category
Development
Specialty
Backend
Experience
4+ years
Tech stack
  • Python
  • Go
  • Kubernetes
  • PyTorch
  • C++
  • Rust
  • LLMs

Location

Work model
Office
Region
United States
Offices
  • San Francisco, United States
  • New York, United States
Relocation
Offered
Visa sponsorship
Sponsored

Compensation

Salary
$300,000 - 400,000 / year
Pay period
Annual

Employment

Type
Full-time

Company

Industry
AI

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason