Senior Software Engineer (Agentic Search) - Web Access Engineer

Description

About Tavily

We're building the infrastructure layer for agentic web interaction at scale. Our API is designed from the ground up to power Retrieval-Augmented Generation (RAG) and real-time reasoning in AI systems. By connecting LLMs to high-quality, trustworthy web content, we help developers build agents that are not only intelligent — but also informed.

We work with some of the most innovative teams in AI — from small startups shaping the ecosystem to the largest enterprises deploying AI at scale. Whether it's powering sales assistants, research copilots, or internal knowledge tools, we're the missing link between LLMs and the real world.

Responsibilities

• We are looking for a Senior Software Engineer to work on the web access infrastructure of a novel search engine tailored for agentic AI consumption.
• In this role, you will focus on making our access to public web content reliable and efficient. You will investigate how modern websites and browsers communicate across the full request lifecycle, find the root cause when a request fails or returns something unexpected, and build the fixes into production systems operating at internet scale. Your work ensures that high-quality content is consistently available to the crawling, indexing and retrieval systems downstream.
• In this position, your responsibility will be to
• Diagnose failed or degraded requests at the right layer: network and IP, TLS and HTTP/2 behavior, headers and session state, client-side scripts, or browser characteristics
• Analyze how websites verify and classify clients, including how client-side scripts and protocol-level behavior shape what a site serves to different clients
• Build clients and tooling that behave consistently and compatibly with standard browsers across all of these layers, using a real browser where a lightweight client is not enough
• Monitor access quality as websites change: detect regressions, find the cause, and measure success on real content rather than on status codes
• Build the tooling to do this at scale: traffic capture and comparison against real browsers, per-site and per-method success tracking, and the cost of each successful fetch
• Choose the fetch method for each host, from a lightweight HTTP client to a full browser, balancing quality against cost
• You may be a good fit if you
• Have 5+ years of experience building production systems, and are an expert in several of the following areas.
• The request lifecycle
• HTTP/1.1, HTTP/2 and HTTP/3, TLS handshakes and TLS fingerprints (JA3/JA4), and header behavior
• What a server can observe at each step of a request, and what a client controls at each step
• A debugging method, not guesses: replaying captured browser traffic, comparing against a real browser on the same network path, and changing one variable at a time
• Client-side analysis
• Reading and analyzing complex or obfuscated JavaScript, and observing what a script measures by hooking browser APIs
• How browser, device and network characteristics are measured (canvas, fonts, WebGL, screen and window metrics) and how consistent they are across layers
• Browser instrumentation and the Chrome DevTools Protocol, used as tools for investigation
• Running it for real
• Have run this in production and can give numbers: requests per day, sites or hosts, and the success rate and how you measured it
• Strong programming in at least one of Go, Python, Rust or C++
• Strong candidates may also have experience with
• Working at a web data provider, a browser or web-security company, or a company that collects data from complex sources
• Contributions to browser networking or fingerprinting-related open-source tooling, or public write-ups of how client verification works
• Building or maintaining a patched browser
• Networks, proxies and IP reputation
• Internet-scale distributed systems, Kubernetes and cloud infrastructure
• The kind of problems you will work on
• A site returns a different response to our client than to Chrome on the same network path. Find out why, and fix that cause.
• A site changes how it verifies clients and success drops across a group of sites. Find what changed, adapt, and make the next change cheaper to catch.
• Switching to a browser-like network profile makes results worse on some hosts. Work out what no longer matches.
• We conduct coding interviews as part of the process.