Software Engineer, Web Crawling

Description

The company is an applied AI lab building a search engine. It runs large-scale infrastructure to crawl the whole web, trains embedding models to process what it finds, and builds high-performance vector databases to query the results. Its search already serves Cursor, Cognition, HubSpot, and over 400,000 developers. The company has raised $350m from Lightspeed, Benchmark, and a16z.

The long-term aim is search over all of the world's information that goes well beyond Google. The team builds large-scale ML systems that will shape how people consume information in the new AI era.

Responsibilities

• Build a distributed crawler that handles 100M+ pages per day.
• Build crawling at Google scale across the entire web.
• Tune crawl politeness and rate limits across thousands of domains.
• Design systems that detect and handle dynamic content, JavaScript rendering, and anti-bot measures.
• Create crawl scheduling and prioritization algorithms that maximise coverage efficiency.

Requirements

• Extensive experience building and scaling web crawlers, or the ability to ramp up very quickly.
• Experience with a high-performance language such as C++ or Rust.
• Familiarity with TypeScript, Playwright, modern web design, and CDP (Chrome DevTools Protocol).
• Comfort optimising a system to an exceptional degree.
• Care about finding high-quality knowledge and understanding how much that matters for the world.

Conditions

Visa sponsorship is available for international candidates, including STEM OPT, OPT, H1B, O1, and E3. The company cannot guarantee a visa, but it has a track record of sponsoring candidates from around the world, and after an offer the team will work hard to secure one.

Equal-opportunity statement not provided in the posting.

Benefits

• Premium healthcare covering medical, dental, and vision.
• Fertility benefits.
• 16 weeks of fully paid parental leave for all new parents.
• A monthly wellness stipend for all employees.