Every LLM call those customers make runs through its code, so a failure there reaches them directly.
You will own key parts of the proxy: routing, auth, caching, spend tracking, guardrails and observability. The aim is correct, fast and dependable behavior at 1B+ requests a month.
Much of the work is tracking down silent failures and fixing root causes in code instead of adding another dashboard. You will also refactor components that keep failing while keeping their behavior unchanged.
The team is mostly early-career, and you will set the standard for how they write, test and ship backend code.
Responsibilities
Take end-to-end ownership of core proxy components, from design through production, including incidents.
Find and eliminate entire bug classes, such as race conditions, stale cache state across replicas, and errors swallowed before users see them.
Make sure every guardrail and policy check, such as applyguardrail, is logged, traceable and fails loudly when it should.
Build guardrails that run inside the gateway and block common attacks on LLM APIs, MCP servers and agents.
Keep Datadog, Splunk, Prometheus and OpenTelemetry integrations accurate at 1B+ requests a month, with little latency overhead and no memory leaks, including from Prometheus metric cardinality.
Help move performance-critical paths from Python to Rust without changing behavior.
Raise the team's standards for code review, testing and debugging.
Requirements
5+ years building production backend systems, with real ownership of mission-critical work such as payments, gateways, core APIs, data stores or anything where downtime hurts customers.
Strong Python skills and experience with async or high-throughput services, such as asyncio or FastAPI.
A record of making production systems more reliable through code, via refactors, re-architectures, migrations or permanent root-cause fixes.
Excellent debugging across services, databases (Postgres, Redis) and distributed systems.
High ownership: you answer for the customer impact of what you ship.
BS or MS in Computer Science or a related field, or equivalent proof of exceptional ability.
Work in production with Rust, Go or C++.
Time at a fast-growing startup with a large, real codebase.
Experience with LLM APIs, proxies or API gateways.