Research Engineer, Index Intelligence

Description

The company is an applied AI lab building a search engine, with infrastructure to crawl the whole web, embedding models to process the content, and vector databases to retrieve from it. It already powers search for Cursor, Cognition, HubSpot and more than 400,000 developers, and has raised $350m from Lightspeed, Benchmark and a16z.

The long-term aim is search over all of the world's information, well beyond Google. The work centers on large-scale ML systems that shape how people will get information in the new AI era.

The web is crawled continuously, but it cannot all be stored, refreshed or processed by a GPU. Decisions are needed on which pages to index, which links to follow, which pages duplicate existing ones, which have gone stale, and which should never have been collected. Those choices matter more than most ranking work. Today they rely on a mix of learned signals and hand-picked thresholds.

The team is hiring a research engineer to own the signals behind these decisions and the decisions themselves.

Responsibilities

• Own the signals that drive crawling, indexing and deduplication decisions, and the decisions themselves.
• Define what the right outcome is for problems that have no ground truth, as the first step.
• Improve parsing on pages where it currently fails, and measure the gain rather than assuming it.
• Build a model that rates page quality, and align the team on a definition of quality strong enough to label training data.
• Model credibility and misinformation: what a page claims, whether its source is reliable, and whether it was written for people or for crawlers.
• Decide whether two documents are the same in meaning or truly different, so duplicates can be removed without dropping pages users would want to see separately.
• Make sure each signal changes a decision and improves search, and work with the crawling, indexing and retrieval teams, which all use your work.

Requirements

• Hands-on experience with ML classifiers, rankers and calibration.
• Strong data judgment at web scale.
• Comfort owning a problem with no ground truth.
• A care for finding high-quality knowledge and an understanding of why that matters.