Senior Observability Platform Engineer - Alerting

Description

Senior Observability Platform Engineer with a focus on Grafana, alerting and SRE.

The role turns telemetry into dashboards, alerts, service-level objectives, incident workflows and dependable operations. It covers telemetry, alert engineering, platform reliability, incident support and helping users adopt observability. The goal is that operators, engineers, security staff, leadership and FinOps teams can use data through governed dashboards, useful alerts and repeatable processes.

Responsibilities

• Run Grafana as the main visualization and alerting tool, including deploying, administering, securing and maintaining it.
• Create and upkeep dashboards for each role: platform, SRE, security operations, application teams, FinOps and leadership.
• Design, tune and maintain alerts for availability, performance, saturation, errors, security events, telemetry gaps, storage problems and SLO breaches.
• Set the rules for alert standards, severity levels, deduplication, grouping, inhibition, routing, escalation, silencing, suppression and maintenance windows.
• Define, implement and maintain SLIs, SLOs, error budgets, service health standards and reliability reporting for onboarded services.
• Build meta-monitoring dashboards that track pipeline throughput, dropped samples, query latency, ingestion rate, storage use, node health, alerting status and identity-provider health.
• Write and optimize queries in PromQL, LogQL, TraceQL, Elastic Query DSL/KQL or other suitable languages, following the approved platform architecture.
• Work with application teams on RED-method dashboards, service-level monitoring, trace-to-log correlation and views that show deployments.
• Work with IAM, security and data-governance staff to set access controls for dashboards, queries and administrative functions.
• Make sure sensitive telemetry is shown and accessed under approved data-handling and least-privilege rules.

Requirements

• A bachelor's degree in computer science, information technology, engineering or a related field. Equivalent experience may be substituted.
• 7+ years in SRE, observability engineering, monitoring engineering, platform operations, DevOps or production support.
• 3+ years of advanced Grafana administration, dashboard development and alerting.
• A strong track record defining and running SLIs, SLOs, error budgets, alerting standards, incident response and post-incident improvement.
• A proven ability to build useful dashboards and alerts from metrics, logs, traces and event data.
• Proficiency in PromQL and/or LogQL, with Elastic Query DSL/KQL strongly desired.
• Experience connecting alerts to ServiceNow, GitLab, webhooks, ticketing systems and enterprise notification workflows.
• Experience with Kubernetes or OpenShift and troubleshooting cloud-native operations.
• Hands-on work with Git, GitLab CI/CD, Grafana provisioning, GitOps and configuration-as-code.
• Desired: Grafana Enterprise experience, including advanced RBAC, reporting, incident-management features and offline licensing in disconnected environments.
• Desired: Experience with Elastic, Prometheus/Mimir, Loki, Tempo, OpenTelemetry and Grafana Alloy.
• Desired: Experience building dashboards for security operations, cloud cost management, OpenCost or executive operational reporting.
• Desired: Experience in air-gapped, classified, DoD, federal or other highly regulated environments.
• Desired: Familiarity with NIST SP 800-53, NIST SP 800-137, ICD 503, RMF/ATO evidence and FIPS constraints.

Conditions

Full-time position. A TS/SCI clearance with polygraph is required to start.

Travel of up to 10%, within the continental US.

The anticipated posting end date is 3/1/2027.

The proposed salary range is $90,300-$189,600. This is the national average for the role, and the final pay depends on location, federal contract labor categories and wage rates, relevant experience, skills, education and certifications.

The role can be performed from more than one location.

Benefits

• Flexible time off, so you can take the time you need.
• Comprehensive benefits including healthcare, wellness, financial, retirement, family support, continuing education and time off.
• Learning resources to help you grow your career.
• Competitive compensation and learning and development opportunities.