Principal Engineer - Data Retrieval

Description

The company is hiring an engineering lead to design and grow the data acquisition systems behind its foundation models. The first focus is code and STEM data for its language models.

You will build systems that find useful sources, crawl the open web at large scale, and send reliable, high-quality data into training pipelines.

This is a hands-on leadership role across distributed systems, crawl infrastructure, and third-party data ingestion. You own the architecture of one platform covering discovery, URL indexing, frontiers, predictive ranking, scheduling, fetching, and ingestion.

Existing infrastructure is the starting point. The goal is to scale it to pretraining volumes and extend it to more model families, data types, and acquisition workflows.

Alongside building, you lead architecture reviews, critical code work, mentoring, and team growth.

The platform is central to the company's AI models and directly affects the reach and quality of training data. The role suits engineers who enjoy hard distributed systems problems and building platforms other teams rely on.

Responsibilities

• Work with storage and scaling teams on frontier storage and queues that handle fast writes, quick lookups, deduplication, and scheduling by quality, diversity, and freshness.
• Build a loop that extracts links from fetched pages and feeds new candidates back into the crawl frontier.
• Create a shared fetching layer that retrieves pages reliably and politely at scale, handles dynamic websites, and keeps resource use in check.
• Build reusable pipelines for open-web, licensed, and partner content covering text, PDFs, code repositories, and multimodal assets. Define stable connector interfaces so teams can add sources without platform changes.
• With parsing and curation teams, keep valuable code, math, and structured material and raise the share of data accepted.
• With legal, privacy, security, and governance partners, turn publisher preferences, opt-outs, license terms, and use limits into auditable controls that travel with the data.
• Own production readiness: capacity planning, monitoring, fault tolerance, recovery, and cost. Instrument the pipeline end to end and set targets for throughput, freshness, reliability, and training deadlines.
• With research and product teams, compare internal crawls and external datasets through matched pilots. Ship reproducible releases that link source coverage and processing quality to controlled training experiments.
• Balance near-term data delivery against long-term platform work and communicate the tradeoffs across teams.

Requirements

• BS, MS, or PhD in computer science, computer engineering, or a related technical field, or equivalent experience.
• 15+ years in software engineering, including designing and running large distributed systems, setting technical direction, and delivering complex infrastructure across several teams.
• Deep experience with web crawling or closely related discovery, indexing, or ingestion systems, including distributed scheduling, stateful services, storage, and high-volume data processing.
• Strong hands-on engineering and system design skills. You move between architecture, implementation, performance analysis, and production debugging.
• Experience with batch or streaming pipelines, including consistency, retries, duplicate processing, backpressure, and recovery. Proven gains in throughput, latency, resource efficiency, or reliability.
• Experience delivering data platforms for research, ML, search, or other demanding users. You turn changing requirements into clear interfaces and measurable outcomes.
• Strong communication and leadership through influence across engineering, product, research, and governance teams.
• Stand out: experience designing a web-scale crawl platform, scaling frontier storage, or merging several acquisition systems into shared infrastructure.
• Stand out: direct experience supplying foundation-model pretraining data, including targeted discovery, quality-aware crawl ranking, and acquisition of code or technical content.
• Stand out: experience implementing publisher consent, training-use opt-outs, dataset licensing controls, or provenance systems at scale.
• Stand out: background in browser-based or agentic data collection, programmable extraction, or sandboxed environments for tool-use data or post-training evaluation.
• Stand out: experience onboarding licensed or multimodal datasets, building reusable partner integrations, or running low-latency streaming feeds for retrieval and grounding.