Staff Software Engineer, Backend (Observability)

Description

The company's Observability team builds the platform engineers use to see how its systems behave and perform.

The platform takes in 200+ TB per day of metrics, logs, and traces from hundreds of services running on more than 80 Kubernetes clusters. When a checkout fails, this data is how engineers learn why.

The team is moving to wide events, where each request produces one richer event, to raise the signal-to-noise ratio and speed up debugging. It is also adding log sampling and building observability practices into agentic development and code review, so AI-generated code arrives well instrumented.

This is a Staff Software Engineer role and the technical lead for the team. You will code most of the time while setting the roadmap, running cross-functional projects, mentoring six engineers, and working closely with infrastructure and product teams.

Responsibilities

• Own the architecture of the telemetry pipeline, balancing reliability, query speed, and cost at 200+ TB per day.
• Design log sampling strategies that cut volume and cost while keeping the data engineers need during incidents.
• Shape how observability fits into agentic development so AI coding agents and code review catch missing instrumentation before code ships.
• Set a year-long technical strategy for the team and connect it to high-impact business projects.
• Work with product management, design, and analytics across the development lifecycle so technical risks and trade-offs are understood and managed.
• Multiply the team's impact by defining and promoting technical solutions and operating processes.
• Keep the team's operations and availability sound through monitoring, triage rotations, playbooks, policies, testing, and alerting that support on-call and routine upkeep.
• Build a culture of quality and ownership by setting code review and design standards, then promoting them beyond the team through writing and tech talks.
• Grow your teammates through feedback, guidance, and by setting an example.

Requirements

• 8+ years designing, building, and launching large-scale backend systems in languages such as Python or Kotlin.
• Deep experience building highly available distributed systems with technologies such as AWS, MySQL, Flink, and Kubernetes.
• Experience building or running high-volume telemetry systems for metrics, logs, or traces, using tools such as OpenTelemetry, Kafka, and Terraform.
• A well-formed view of modern observability practice, including structured and wide events, sampling, cardinality management, and SLOs.
• Experience getting many engineering teams to adopt a platform standard through tooling and outreach.
• Experience delivering major features or components, or retiring existing functionality, through a written technical and execution plan. You write clear, maintainable code that others can understand and use.
• Comfortable with ambiguity and able to move from low-level language details to the architecture of large systems.
• A record of collecting and acting on feedback from engineering and cross-functional peers.
• Strong verbal and written communication for working with a global engineering team.
• Equivalent practical experience or a Bachelor's degree in a related field.

Conditions

Base pay grade P and equity grade 13.

New hires usually start near the bottom of the pay range. Pay is set by location, experience, and job-related skills.

USA base pay range in CA, WA, NY, NJ, or CT: $230,000 to $290,000.

USA base pay range in all other U.S. states: $204,000 to $264,000.

Remote-first. Most roles can be done from almost anywhere in the country of employment. Some roles may occasionally need in-person work at an office, and a few are office-based. All new hires are invited to an in-person onboarding.

For positions based in San Francisco or Los Angeles, the company considers qualified applicants with arrest and conviction records, as required by law.

The company runs an inclusive interview process and provides accommodations for candidates with disabilities.

Benefits

• Base pay is part of a total compensation package that may include equity, monthly stipends, and benefits.
• Medical, dental, and vision coverage for you and your dependents, with medical coverage subsidized at 100%.
• Monthly stipends for your tech setup, plus health and wellness options that suit you.
• Flexible time off and generous holidays to help you rest.
• Employee stock purchase plan (ESPP) that lets you buy company stock at a discount.