Principal Engineer - Data Retrieval

Description

The company needs an engineering lead to design and grow the data acquisition systems behind its foundation models.

The first focus is code and STEM data for one model line, followed by systems that find valuable sources, crawl the open web at scale, and send reliable, high-quality data to training pipelines.

The role combines distributed systems, crawl infrastructure, and third-party data ingestion, and it is hands-on technical leadership.

You will own the architecture of one platform covering source discovery, URL indexing, crawl frontiers, predictive ranking, scheduling, distributed fetching, and ingestion.

Existing infrastructure is the starting point. The goal is to bring it to pretraining scale and extend it to more model families, modalities, and acquisition workflows over time.

You will also lead architecture reviews and critical implementation work, mentor engineers, and help grow the team.

Responsibilities

• Alongside storage and scaling teams, design frontier storage and queueing with fast writes, efficient lookups, deduplication, and scheduling by quality, diversity, and freshness.
• Build a continuous discovery loop that pulls links from fetched pages and feeds new candidates into the crawl frontier.
• Create a shared crawl execution layer that fetches reliably and politely at scale, handles dynamic websites, and keeps resource use in check.
• Build extensible pipelines for open-web, licensed, and partner content, including text, PDFs, code repositories, and multimodal assets, with stable connector interfaces so teams can add sources without rebuilding the platform.
• Work with parsing and curation teams to keep valuable code, math, and structured content and raise the share of accepted data.
• With legal, privacy, security, and data governance partners, turn publisher preferences, opt-outs, licensing terms, and use restrictions into auditable controls that travel with the data downstream.
• Own production readiness, including capacity planning, observability, fault tolerance, recovery, and cost efficiency. Add monitoring across the whole pipeline and define targets for throughput, freshness, reliability, and hitting training deadlines.
• With research and product teams, assess internal crawls and external datasets through comparable pilots, and ship reproducible data releases tied to controlled training experiments.
• Coordinate across teams and make clear tradeoffs between immediate data delivery and long-term platform work.

Requirements

• Hold a BS, MS, or PhD in computer science, computer engineering, or a related technical field, or equivalent experience.
• Have 15+ years of software engineering experience, including designing and running large-scale distributed systems, setting technical direction, and delivering complex infrastructure across several teams.
• Have worked deeply on web crawling or similar systems for finding, indexing, or ingesting data, covering scheduling across machines, stateful services, storage, and high-volume processing.
• Be strong in hands-on engineering and system design, moving between architecture, implementation, performance analysis, and production debugging.
• Have built batch or streaming pipelines and can reason about consistency, retries, duplicate processing, backpressure, and recovery, with a record of improving throughput, latency, resource efficiency, or operational reliability.
• Have delivered data platforms for research, ML, search, or other demanding technical users, and turned changing requirements into clear interfaces and measurable outcomes.
• Communicate clearly and lead through influence across engineering, product, research, and governance teams.
• Stand out: have architected a web-wide crawl platform, scaled crawl frontier storage, or merged several acquisition systems into shared infrastructure.
• Stand out: have supplied foundation-model pretraining data, including targeted discovery, quality-aware crawl ranking, or acquisition of code or technical content.
• Stand out: have implemented publisher consent, training-use opt-outs, dataset licensing controls, or provenance infrastructure at scale.
• Stand out: have worked with browser-based or agentic data collection, programmable extraction, or sandboxed environments for tool-use data or post-training evaluation.
• Stand out: have onboarded licensed or multimodal datasets, built reusable partner integrations, or delivered low-latency streaming feeds for retrieval and grounding.

Conditions

Base salary is set by location, experience, and pay for similar roles, within 272,000 USD to 431,250 USD.

This is an existing vacancy, and applications are accepted at least until October 30, 2026.

The company uses AI tools in its recruiting process.

The company is an equal opportunity employer and does not discriminate on the basis of any characteristic protected by law.