The company's open-source AI gateway is the most widely used of its kind. Large firms including Adobe, Netflix, NASA and NVIDIA send their LLM traffic through its code, so a defect reaches them directly.
The work centres on the proxy's routing, auth, caching, spend tracking, guardrails and observability, which must behave correctly and quickly at 1B+ requests a month.
You'll be on the critical path for some of the largest companies in the world and will have real ownership of core systems on a small, senior-leaning team.
Responsibilities
Take full ownership of core proxy components, from design through production, including their incidents.
Eliminate whole categories of defects such as race conditions, stale cache state across replicas, and errors swallowed before users see them.
Make sure each guardrail and policy check, including applyguardrail, is logged, traceable and raises errors visibly when it should.
Build guardrails that run inside the gateway and stop common attacks on LLM APIs, MCP servers and agents.
Keep Datadog, Splunk, Prometheus and OpenTelemetry integrations accurate at 1B+ requests a month, with little added latency and no leaks such as runaway Prometheus metric cardinality.
Help move performance-critical paths from Python to Rust without changing behavior.
Set the team's standards for code review, testing and debugging.
Requirements
2+ years of backend work in production, including real ownership of a mission-critical system such as payments, gateways, core APIs, data stores, or any service where downtime hurts customers.
Strong Python and experience with async or high-throughput services, using asyncio, FastAPI or similar.
A proven record of improving production reliability through code, via refactors, re-architectures, migrations, or permanent root-cause fixes.
Excellent debugging across services, Postgres and Redis databases, and distributed systems.
High ownership: you answer for the customer impact of what you ship.
BS/MS in Computer Science or a related field, or equivalent proof of exceptional ability.
Rust, Go or C++ used in production.
Experience at a fast-growing startup with a large, real codebase.
Experience with LLM APIs, proxies or API gateways.