Sr. Observability Systems Engineer

Description

This is a senior individual contributor role with significant architectural ownership at the company, which runs data centers and detects, correlates, and responds to infrastructure incidents across them.

You will set the technical direction for that work and mentor engineers on the team as the platform grows.

The role suits someone who enjoys making complex distributed systems observable, is willing to make architecture decisions that last beyond their own time, and wants to apply AI and agentic automation to infrastructure operations.

Responsibilities

• Set technical direction and standards for observability across the organization: instrumentation conventions, alerting approach, and dashboard patterns.
• Mentor Observability team engineers and advise SRE, Storage Engineering, and Network Engineering on observability practice.
• Lead design reviews for observability work and speak for the team in cross-functional architecture discussions.
• Assess whether to build or buy observability and AIOps tooling, weighing data sovereignty and compliance.
• Write and maintain design documents, runbooks, and architectural decision records so future engineers can operate and extend the platform.
• Own and evolve the metrics pipeline: VictoriaMetrics, Prometheus, and the Consul-based service discovery that keeps scrape targets current as infrastructure grows.
• Own and extend the logging stack (OpenSearch, Logstash, Filebeat) across all data centers.
• Design and expand distributed tracing with Jaeger and OpenTelemetry to cover new services beyond current application instrumentation.
• Set Grafana dashboard standards for infrastructure, storage, and application performance.
• Work with application teams to expand internal telemetry and improve visibility.
• Define capacity and retention strategy for the telemetry stack as the fleet grows.
• Write infrastructure-as-code for observability deployments across multiple Kubernetes clusters.
• Write and review automation playbooks for configuration management on Debian-based bare metal servers.
• Work with the SRE team on Consul-based service discovery that automates VictoriaMetrics scrape targets.
• Lead Ceph storage observability, covering OSD health, performance metrics, and alerting integration.
• Work with Network Engineering on BGP, routing, and network-level observability.

Requirements

• 8+ years in systems engineering, SRE, DevOps, or observability, including proven technical leadership on a platform or infrastructure team.
• Hands-on experience running Prometheus, VictoriaMetrics, or similar metrics platforms at scale, including PromQL and Alertmanager.
• Experience designing and running log pipelines such as Elasticsearch/OpenSearch, Logstash, Filebeat, or similar.
• Working knowledge of distributed tracing (Jaeger, OpenTelemetry, or equivalent), including how to decide what to instrument.
• Proficiency in Python and/or Go for automation and tooling.
• Experience deploying workloads on Kubernetes, with an understanding of pod networking and storage.
• Experience with infrastructure-as-code and configuration management tools.
• Strong Linux internals knowledge, Debian or Ubuntu preferred.
• Track record of mentoring engineers and raising the technical bar on a team.
• Excellent written and verbal communication. You will work across engineering, SRE, security, and leadership, and write design documents that drive decisions.
• Preferred: experience evaluating or deploying AIOps tooling for alert correlation, deduplication, or automated root cause analysis. This is a strong differentiator.
• Preferred: experience building, deploying, or operating AI agents or agentic AI systems.
• Preferred: familiarity with LLM APIs (Anthropic, OpenAI) or self-hosted LLM inference in infrastructure settings.
• Preferred: familiarity with Ceph, including OSD management, pool configuration, and CRUSH maps.
• Preferred: experience with incident management platforms.
• Preferred: familiarity with BGP, network observability, or multi-data-center environments.
• Preferred: experience with secrets management in production.
• Preferred: experience in a regulated or compliance-driven environment.
• Preferred: prior work at a storage-focused company or hyperscale infrastructure environment.

Conditions

Base pay ranges are shared for all US-based postings, regardless of state. Final offers depend on factors such as candidate location, skills, depth of experience, and relevant licenses or credentials.

The expected base salary range is $178,000 to $200,000, plus a 10% target annual bonus and RSUs.

The company is an Equal Opportunity Employer and welcomes candidates of every background.