You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Cohere

Staff Software Engineer, Data Infrastructure

  • Remote
  • 6+ years

Salary

$245,000 - 335,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company builds security-first foundation AI models and end-to-end products for enterprises, and trains and deploys frontier models for businesses building their own AI systems.

The company is global, headquartered in Toronto, with teams in London, New York City, San Francisco, Montreal, Paris, Berlin and Seoul.

The Data Infrastructure team builds the unified storage system that supplies data and model checkpoints to every training run, at petabyte scale.

The system must deliver data fast enough to keep thousands of GPUs busy across several training clusters, and you would design it from the ground up.

You would be a key contributor and lead the effort on a problem few teams have faced at this scale.

Responsibilities

  • Design, build and run the distributed storage system that serves model training and evaluation, guiding it from concept to production.
  • Lead the rollout of the system across several Kubernetes clusters at petabyte scale, and set the standards and practices the organization follows.
  • Work with researchers and training-infrastructure teams to learn how jobs read and write data, then turn those patterns into throughput, latency and durability requirements for the roadmap.
  • Solve networking, I/O and consistency problems in moving large datasets and checkpoints across regions and storage backends, measuring success by GPU idle time and time-to-insight.

Requirements

  • You may be a good fit if you have strong storage fundamentals, such as replication, consistency, caching and data lifecycle management, plus proven architecture and design experience.
  • You bring strong Python and Go skills, can lead implementations and mentor others in either language, and are willing to pick up the other.
  • You have run stateful systems on Kubernetes, working with Persistent Volumes, CSI drivers and StatefulSets, and can plan and carry out operational strategies.
  • You have hands-on experience with cloud object storage such as S3 and with POSIX-style filesystems.
  • Bonus: experience with parallel or HPC filesystems such as Weka, VAST or Lustre, and a record of running them at scale.
  • Bonus: familiarity with the data-loading and checkpointing patterns used in large-scale model training, and the ability to improve on them.
  • Bonus: interest in how large models are trained and evaluated, including the trajectory and eval data those runs produce, and the ability to help others understand the workloads the platform serves.

Conditions

The role is remote-friendly. Offices are in Toronto, London, New York City, San Francisco, Montreal, Paris, Berlin and Seoul, with more planned.

Office-based staff get a daily lunch program, snacks, and regular community and social events.

Remote staff who live far from an office can use a co-working benefit to work alongside others in their city.

The company may use AI-enabled tools to screen applicants against the role's criteria, to help recruiters find likely candidates, though recruiters may still review any application.

The company is committed to equal opportunity and welcomes applicants from all backgrounds.

Benefits

  • Weekly lunch stipend of $75 or £75, or the equivalent in your local currency.
  • Health and dental coverage, with a separate budget for mental health.
  • RRSP matching, a 401K plan and a pension scheme.
  • 100% parental leave top-up for up to 6 months, available to either parent.
  • Annual enrichment budget for arts and culture, fitness and wellness, quality time, and a workspace improvement credit.
  • Education and learning stipend for conferences, courses and coaching.
  • 6 weeks of paid vacation, which is 30 working days.
  • Travel budget to visit other offices if you are remote, plus an annual company offsite.
  • $500 home office stipend to set up your workspace.

Where you’d work

Fully remote

You can work from

  • United States

About the company

Cohere

  • Industry: AI

Also hiring in London, United Kingdom, New York, United States, Ottawa, Canada and 6 more places

20 of their 27 open roles are remote

Your chances

Worth a look before you spend an evening tailoring a CV for it.

  • 21 checks run
  • 1 red flag

Still hiring?

14 checks

No red flags

How crowded?

7 checks

1 red flag

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details12 facts · Role, Location, Compensation, Employment, Company
Tech stack
  • Python
  • Kubernetes
Seniority
Staff
Type
Full-time
Industry
AI
Specialty
Data Engineering
Region
United States
Show 6 more factsShow less

Role

Category
Data & Analytics
Specialty
Data Engineering
Seniority
Staff
Experience
6+ years
Tech stack
  • Python
  • Kubernetes

Location

Work model
Remote
Region
United States
Remote from
  • United States

Compensation

Salary
$245,000 - 335,000 / year
Pay period
Annual

Employment

Type
Full-time

Company

Industry
AI

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason