Senior DevOps Engineer, Observability and FinOps

Description

This role makes mission systems observable and tracks what they cost, on networks that have no route to the internet.

You will split your time between platform engineering on Kubernetes, OpenShift, infrastructure as code and CI/CD, and full stack work on the dashboards, APIs and small internal tools that operators and finance staff use every day.

Responsibilities

• Stand up and run the observability stack on each network, such as Grafana Alloy, Mimir, Loki and Tempo on OpenShift, or Elastic or Dynatrace where those have been selected.
• Add OpenTelemetry and Prometheus exporters to applications and infrastructure, and set consistent labels, tags and names.
• Create live dashboards that show mission status and service health for operators, engineers and leadership.
• Set up cost visibility and allocation across AWS, OpenShift and on-premises resources using tagging standards and showback or chargeback models.
• Build cost and utilization dashboards that combine billing exports with observability metrics, showing unit cost per mission application, namespace and program.
• Find and carry out savings through rightsizing, removing idle resources, storage lifecycle rules, and analysis of Reserved Instance and Savings Plan coverage.
• Automate infrastructure and stack deployment with Terraform, Ansible, and Helm or Operators, and manage configuration through GitOps with Argo CD or similar.
• Build and maintain CI/CD pipelines in GitLab CI, Jenkins or Tekton, including image scanning, signing, and promotion into air-gapped registries.
• Package software and its dependencies for offline transfer and cross-domain delivery.
• Develop lightweight web apps, APIs, and Grafana plugins or panels for custom status and cost views.
• Write Python or Go integration scripts that pull data from billing exports, CMDBs and ticketing systems.
• Write runbooks, architecture diagrams and user guides, train operators, and mentor junior engineers.

Requirements

• Hold a TS/SCI clearance with a polygraph.
• Hold a bachelor's degree in computer science, information technology, software engineering or a closely related field.
• Have at least 3 years deploying and running one enterprise stack: Grafana with Prometheus, Elastic (ELK), Dynatrace or Splunk.
• Have instrumented metrics, logs and traces, with working knowledge of OpenTelemetry or Prometheus exporters.
• Have built operational dashboards and alert rules that real operators use, and understand SLIs and SLOs.
• Have at least 3 years running Kubernetes in production, with Red Hat OpenShift a strong fit.
• Have at least 2 years on AWS, including EC2, S3, EKS, IAM, CloudWatch and Cost and Usage Reports.
• Have a proven record of analyzing cloud spend, applying tagging strategies, and delivering at least one cost optimization result.
• Have working skill with Terraform, Ansible, and Helm charts or Operators.
• Have built and maintained pipelines in GitLab CI, Jenkins or an equivalent tool, and are comfortable with Git workflows.
• Be proficient in Python or Go and Bash scripting, and able to write and query SQL and PromQL or equivalent.
• Have RHEL administration and troubleshooting experience.
• Have hardened systems to DISA STIGs, and be familiar with RMF and NIST SP 800-53 controls.
• Have worked on air-gapped or disconnected networks, including offline package and image management.
• Desired: hands-on use of Grafana Alloy, Mimir, Loki and Tempo at scale.
• Desired: production experience with two or more of Grafana, Elastic and Dynatrace, allowing side-by-side judgment.
• Desired: experience with store-and-forward pipelines, edge collectors, or Kafka or similar buffering over intermittent links.
• Desired: experience moving data or software through cross-domain solutions or other approved transfer processes.
• Desired: experience with OpenCost or Kubecost, CloudHealth, Apptio Cloudability, or AWS Cost Explorer and Cost and Usage Report analytics.
• Desired: experience with Istio or a comparable service mesh, including mesh-level metrics and tracing.
• Desired: experience with cluster monitoring and user workload monitoring, Operators, and OpenShift GitOps and Pipelines.

Conditions

The minimum clearance needed to start is TS/SCI with polygraph.

This is a full-time, regular position.

Travel is up to 10%, within the continental US.

The anticipated posting end date is 11/13/2026.

The proposed salary range is $98,500 to $206,800. Because the role can be based in more than one location, the range shown is the national average. Final pay depends on location, Federal Government contract labor categories and wage rates, prior experience, specific skills, education and certifications.

Benefits

• Comprehensive benefits including healthcare, wellness, financial, retirement, family support, continuing education and time off.
• Flexible time off that lets you take the time you need.
• Access to learning resources to support your career goals.
• Competitive compensation.