The role is about building large-scale web crawling and data acquisition systems that feed AI research and model development.
You will work with research teams to produce high-quality web datasets used for both training and inference.
Responsibilities
Build and run crawlers that cover social media, travel, and multilingual sites.
Create high-throughput, fault-tolerant collection systems that handle millions to billions of URLs daily.
Adjust crawling methods for anti-bot defenses, rate limits, and sites that rely heavily on JavaScript.
Build large-scale pipelines that clean, deduplicate, filter, and normalize web data.
Work with research teams to shape datasets for research and model training.
Track crawl speed, coverage, and data quality, and adapt to changes in the web.
Tune infrastructure for cost, latency, and reliability across cloud and bare-metal setups.
Requirements
Usually 4 or more years building web crawlers or large-scale data acquisition systems. A PhD may bring this down to 2 or more years. Hands-on crawler and pipeline work is essential.
Experience running distributed data pipelines at terabyte scale or larger, ideally handling millions to billions of URLs per day.
Strong skill in Go, Rust, Python, Java, or C++, with a solid grasp of distributed systems and parallel processing.
Deep understanding of HTTP, networking, and browser behavior, including handling anti-bot systems, rate limits, and dynamic sites.
Hands-on use of headless browsers such as Playwright or Puppeteer, Chrome DevTools Protocol, proxy systems, or IP rotation.
Experience with web data quality, benchmarking, cleaning, deduplication, and normalization. NLP pipelines or ML dataset curation are a plus.
Experience running crawling or data workloads on cloud platforms or bare-metal infrastructure. A degree in computer science, engineering, or a related field is a plus.
Conditions
Base salary ranges from $160,000 to $250,000 per year.
Visa sponsorship is not available.
Remote position.
Where you’d work
Fully remote
You can work from
United States
No visa sponsorship
Good
Your chances
Still hiring, not crowded yet, and you'd be among the first.
17 checks run
5 good signs
1 red flag
Still hiring?
11 checks
Actively hiring
StaleHiring
In its favour3
Still on the company's own careers site, checked 1 h agoModerate evidence
Found in the last 48 hours, before the big job boardsModerate evidence
States its salarySlight evidence
Against it1
Listed for the 4th time after earlier listings of it closedModerate evidence
How crowded?
6 checks
Low
QuietCrowded
In its favour2
In its Early Window: not on the big job boards yetStrong evidence
Remote within United States onlySlight evidence
?
Fits Me
How well does this role fit you?
Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.