Research Engineer, Content Understanding

Description

The company is an applied AI lab building a search engine. It builds large-scale infrastructure to crawl the whole web, trains embedding models to process what it collects, and builds high-performance vector databases to query it.

The company already runs search for Cursor, Cognition, HubSpot and over 400,000 developers, and has raised $350m from Lightspeed, Benchmark and a16z.

The long-term aim is search over all of the world's information, well beyond Google. The team builds ML systems at massive scale that will shape how people consume information in an AI-driven world.

Search quality depends on how well the system understands each page. Before a page can be retrieved, the system must work out what it says: separate content from surrounding clutter, classify its type and topic, decide whether it is usable, extract its publication date, rate its quality and trustworthiness, and check whether it adds anything beyond pages already indexed. This has to work on every page on the web, in any language and any format.

Part of this is established document understanding. The rest is open. Credibility and misinformation, AI-generated and machine-spun text, and pages built to be found rather than read have no settled answers, and search results are only as trustworthy as those answers.

Responsibilities

• Make parsing work on pages where it currently fails, and measure the improvement rather than assume it.
• Train a model to judge page quality, and align the team on a definition of quality clear enough to label against.
• Model credibility and misinformation: what a page claims, whether its source is reliable, and whether it was written for readers or for crawlers.
• Determine whether two documents are semantically the same or meaningfully different, so the web can be deduplicated without merging pages users would want to see separately.
• Build classification and extraction that are accurate at web scale and cheap enough to run across all of it.
• Design supervision for problems that have no labels yet, and find out whether they are learnable.
• Trace a bad search result back to the page-level prediction behind it, and fix it at the source.
• Projects are flexible and chosen based on your skills and interests.

Requirements

• Graduate-level ML experience: a Master's or PhD with at least 2 years of relevant experience, or an exceptionally strong undergraduate background.
• Ability to build a transformer from scratch in PyTorch.
• Experience training models that then had to be cheap enough to run everywhere.
• Enjoy building large datasets and working closely with the data, since most gains come from supervision rather than architecture.
• Comfortable working on problems where the ground truth does not exist yet and defining it is part of the job.
• Care about finding high-quality knowledge and understand why it matters.