You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Thinking Machines Lab

Software Engineer, Tinker Infrastructure

  • Office
  • 6+ years

Salary

$300,000 - 400,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company trains frontier models, builds a fine-tuning API, and works on interfaces for human-AI communication, with a stated focus on AI that extends human judgment.

The fine-tuning platform lets researchers and developers customize open-weights models using their own data and algorithms, while the company runs the underlying infrastructure. It is adding customers, features, and use cases quickly, and it runs as a live production service that never goes down.

This role helps build and scale the distributed systems behind that platform. You will work with research and infrastructure teams to turn new training and sampling workloads into systems that are fast, correct, and resilient. The work covers how jobs are placed, checkpointed, recovered, and monitored across thousands of accelerators running at once, and real users depend on it staying up.

It is a hands-on role with substantial ownership. You will make architectural decisions, lead incidents to resolution, and set technical direction for parts of the platform with little day-to-day oversight.

Responsibilities

  • Design, build, and run the distributed systems behind the platform, covering job scheduling, resource orchestration, checkpointing, and fault recovery on large multi-node GPU clusters.
  • Own the reliability, latency, and throughput of live production systems that serve external users running concurrent training and sampling jobs.
  • Design for graceful degradation and quick recovery so that node failures, preemptions, and network partitions do not lose user progress or data.
  • Improve observability with metrics, tracing, and alerting that catch and diagnose problems before or as they reach users.
  • Lead incident response for the systems you own, run root-cause analysis, and turn what you find into durable fixes.
  • Work with research and ML infrastructure teams to understand new training and sampling workloads and evolve the architecture to handle them at scale.
  • Make build-versus-buy and architectural trade-off decisions for core infrastructure, and mentor other engineers on distributed systems practice.

Requirements

  • Minimum qualifications: 7+ years of experience building, running, and scaling distributed systems in production, including being directly on call for live services.
  • Minimum qualifications: a solid grasp of distributed systems fundamentals such as consensus, consistency models, failure detection, scheduling, and state management under partial failure.
  • Minimum qualifications: a record of designing systems that run reliably at scale and of resolving production incidents under time pressure.
  • Minimum qualifications: strong systems-level coding skills in a language such as Python, Go, Rust, or C++, and comfort working close to the infrastructure layer.
  • Preferred: experience running large GPU or accelerator clusters, including job orchestration, resource scheduling, or ML training infrastructure.
  • Preferred: familiarity with the training or inference stack for large language models and with the systems needs specific to fine-tuning or post-training.
  • Preferred: experience with checkpointing, distributed storage, or high-throughput networking such as RDMA or NCCL in performance-critical settings.
  • Preferred: experience building developer-facing infrastructure or APIs where both reliability and usability matter.
  • Preferred: a record of driving platform-wide architecture decisions and working with significant autonomy in a fast-changing environment.

Conditions

Location: based in San Francisco, California or New York, New York.

Compensation: depending on background, skills, and experience, the expected annual salary is $300,000 to $400,000 USD.

Visa sponsorship: the company sponsors visas and will work through the process with candidates who are a strong fit, though success cannot be guaranteed for every candidate or role.

The company welcomes applicants regardless of background.

Benefits

  • Generous health, dental, and vision coverage.
  • Unlimited PTO.
  • Paid parental leave.
  • Relocation support as needed.

Where you’d work

From the office

The office

Relocation offered

Visa sponsored

About the company

Thinking Machines Lab

  • Industry: AI

Office in San Francisco, United States

Also hiring in New York, United States

Your chances

Worth a look before you spend an evening tailoring a CV for it.

  • 23 checks run
  • 3 red flags

Still hiring?

14 checks

No red flags

How crowded?

9 checks

3 red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details13 facts · Role, Location, Compensation, Employment, Company
Tech stack
  • Python
  • Go
  • C++
  • Rust
  • LLMs
Type
Full-time
Industry
AI
Specialty
Backend
Region
United States
Pay period
Annual
Show 7 more factsShow less

Role

Category
Development
Specialty
Backend
Experience
7+ years
Tech stack
  • Python
  • Go
  • C++
  • Rust
  • LLMs

Location

Work model
Office
Region
United States
Office
  • San Francisco, United States
Relocation
Offered
Visa sponsorship
Sponsored

Compensation

Salary
$300,000 - 400,000 / year
Pay period
Annual

Employment

Type
Full-time

Company

Industry
AI

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason