Research Crawling Engineer

Description

You will join a data infrastructure team that is growing. The team builds large web data systems used in AI research and model development.

Your work is to build dependable crawling and data collection tools that gather data from many kinds of websites. Research teams use that data to make training and evaluation sets for advanced models.

Responsibilities

• Create and run web crawlers for social, travel, multilingual, and other kinds of sites.
• Build collection systems that handle throughput and failures well, processing from millions to billions of URLs daily.
• Handle anti-bot defenses, rate limits, and sites that rely heavily on JavaScript or render content dynamically.
• Construct pipelines that clean, remove duplicates from, filter, and standardize web data at terabyte to petabyte volumes.
• Coordinate with research teams so collection and dataset curation match what models need.
• Track crawl reach, speed, and data quality, and update systems when websites change.
• Reduce cost, latency, and downtime across cloud and bare-metal environments.

Requirements

• You have at least 2 years of experience building web crawlers or large data pipelines. Four or more years is preferred, though a PhD can offset less experience.
• You have run data pipelines at terabyte scale or larger, using distributed or parallel processing.
• You program well in Go, Rust, Python, Java, or C++.
• You understand HTTP, networking, and browser behavior in depth, and you know how to get past anti-bot systems, rate limits, and dynamic sites.
• You are comfortable with headless browsers such as Playwright or Puppeteer, proxy systems, and orchestrating requests.
• Experience evaluating data quality at scale, working with NLP pipelines, or curating machine learning datasets is a plus.
• You have run data workloads on cloud platforms or bare-metal infrastructure.

Conditions

Base salary ranges from $160,000 to $250,000 per year.

Visa sponsorship is not available.

This role is fully remote.