You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Early Window: Closes in
Company hidden

Research Crawling Engineer

  • Remote

Salary

$160,000 - 250,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The role is about building large-scale web crawling and data acquisition systems that feed AI research and model development.

You will work with research teams to produce high-quality web datasets used for both training and inference.

Responsibilities

  • Build and run crawlers that cover social media, travel, and multilingual sites.
  • Create high-throughput, fault-tolerant collection systems that handle millions to billions of URLs daily.
  • Adjust crawling methods for anti-bot defenses, rate limits, and sites that rely heavily on JavaScript.
  • Build large-scale pipelines that clean, deduplicate, filter, and normalize web data.
  • Work with research teams to shape datasets for research and model training.
  • Track crawl speed, coverage, and data quality, and adapt to changes in the web.
  • Tune infrastructure for cost, latency, and reliability across cloud and bare-metal setups.

Requirements

  • Usually 4 or more years building web crawlers or large-scale data acquisition systems. A PhD may bring this down to 2 or more years. Hands-on crawler and pipeline work is essential.
  • Experience running distributed data pipelines at terabyte scale or larger, ideally handling millions to billions of URLs per day.
  • Strong skill in Go, Rust, Python, Java, or C++, with a solid grasp of distributed systems and parallel processing.
  • Deep understanding of HTTP, networking, and browser behavior, including handling anti-bot systems, rate limits, and dynamic sites.
  • Hands-on use of headless browsers such as Playwright or Puppeteer, Chrome DevTools Protocol, proxy systems, or IP rotation.
  • Experience with web data quality, benchmarking, cleaning, deduplication, and normalization. NLP pipelines or ML dataset curation are a plus.
  • Experience running crawling or data workloads on cloud platforms or bare-metal infrastructure. A degree in computer science, engineering, or a related field is a plus.

Conditions

Base salary ranges from $160,000 to $250,000 per year.

Visa sponsorship is not available.

Remote position.

Where you’d work

Fully remote

You can work from

  • United States

No visa sponsorship

Your chances

Still hiring, not crowded yet, and you'd be among the first.

  • 17 checks run
  • 5 good signs
  • 1 red flag

Still hiring?

11 checks

Actively hiring

In its favour3

  • Still on the company's own careers site, checked 1 h agoModerate evidence
  • Found in the last 48 hours, before the big job boardsModerate evidence
  • States its salarySlight evidence

Against it1

  • Listed for the 4th time after earlier listings of it closedModerate evidence

How crowded?

6 checks

Low

In its favour2

  • In its Early Window: not on the big job boards yetStrong evidence
  • Remote within United States onlySlight evidence

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details10 facts · Role, Location, Compensation, Employment
Tech stack
  • Python
  • Go
  • Java
  • JavaScript
  • C++
  • Rust
Type
Full-time
Specialty
Backend
Region
United States
Pay period
Annual
Show 5 more factsShow less

Role

Category
Development
Specialty
Backend
Tech stack
  • Python
  • Go
  • Java
  • JavaScript
  • C++
  • Rust

Location

Work model
Remote
Region
United States
Remote from
  • United States
Visa sponsorship
Not sponsored

Compensation

Salary
$160,000 - 250,000 / year
Pay period
Annual

Employment

Type
Full-time

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason