Data Platform Engineer

Description

The company runs a specialty care platform that connects members with top specialists for surgery, cancer care, infusions and more, through a national network it curates. A dedicated care team, including Care Advocates and nurses, supports members through their care. The company serves more than 6 million members for some of the largest employers in the country.

The data organization is shifting from a central model, where the platform team builds most pipelines, to one where domain teams (Data Engineering, Data Ops, Analytics Engineering) build and own their data products. This role builds and runs the self-serve infrastructure, governance automation, and access controls those teams rely on, instead of writing domain pipelines itself.

This is a hands-on role that administers the Databricks platform and the Lakebase (managed Postgres) estate, and carries the elevated privileges that come with it. You enforce standards through platform controls, automation, and review across Azure infrastructure and IAM, CI/CD, observability, and the access, audit, and lineage layer. You are also the first point of contact for domain teams, from project kickoffs through day-to-day help, and you work closely with the Cloud Platform, Azure SQL DBA, and Security teams on shared foundations. Domain teams own their data products, and the data architecture committee owns the standards.

Hybrid: at least 3 days per week in the Dallas, TX office.

Responsibilities

• Administer Azure Databricks and Unity Catalog against the committee's standards: workspaces, metastore, catalog and schema structure, cluster policies, and compute spend. Provide spend-by-team reporting and chargeback.
• Raise issues with standards owners when a standard blocks teams or no longer matches how the platform is used.
• Build self-serve provisioning workflows, templates, and access patterns so domain teams can create and manage their own data products without waiting on the platform team.
• Act as DBA for the Lakebase Postgres estate: provision and decommission instances, manage users and roles, configure connections, tune queries and indexes, handle version upgrades, backups, point-in-time recovery, and database incidents. Maintain the sync patterns between Lakebase and the lakehouse.
• Build and maintain Terraform infrastructure-as-code for Azure resource groups, networking and private endpoints, Entra ID app registrations, service principals, and Key Vault, so changes are reviewable, repeatable, and auditable.
• Implement and enforce the approved access model across the lakehouse and Postgres: Unity Catalog grants, OBO authentication, ABAC, row-level security, data masking, and role-based access for engineering, ops, and analytics users.
• Build audit logging, lineage capture, and sensitivity tagging so decentralized ownership stays auditable. Build the custom event-sourced audit layer where Databricks cannot capture app-layer events, and respond when it fails.
• Run observability across Databricks, Lakebase, and supporting Azure services, including Datadog integration, metrics and log export, alerting, and SLOs.
• Run infrastructure lifecycle work: provisioning, decommissioning, backup and restore validation, and disaster recovery testing for platform and database workloads.
• Support CI/CD for domain teams: pipelines for notebooks, jobs, DLT pipelines, dbt projects, Databricks Asset Bundles, and Lakebase schema migrations.
• Maintain the GitHub and GitHub Actions integration, including repo structure, branch policies, and CI/CD workflows that domain teams build on.
• Work with the Cloud Platform team on shared Azure networking, Entra ID, Key Vault, policy, and cloud cost. Agree where platform ownership ends, and route Azure infrastructure issues to the right owner.
• Coordinate with the Azure SQL DBA team, which runs the Azure SQL estate separately. Share practice on backup, HA/DR, and PHI handling, and advise on integration points and workload placement.
• Keep a recurring liaison with Security, InfoSec, and Core Apps on cross-team dependencies, and make sure platform capabilities meet HIPAA, SOC 2, and other regulatory requirements.
• Join domain-team project kickoffs early to advise on platform capabilities, access patterns, and the standards a design must meet.
• Be the escalation point for domain engineers blocked on cluster policies, permissions, database performance, or broken CI steps.
• Produce platform KPI reports on usage, uptime, deployment velocity, and spend. Contribute to the platform roadmap and backlog.
• Record reusable patterns from kickoffs, incidents, and liaison work in a shared knowledge base.

Requirements

• At least 4 years in a platform or infrastructure engineering role supporting a data or analytics organization.
• Deep hands-on experience with Databricks (workspace administration, Unity Catalog, Delta Lake) and Azure (Entra ID, networking, Key Vault, and Terraform or Bicep for infrastructure-as-code).
• Hands-on PostgreSQL administration, as you will be the DBA of record: users and roles, connection configuration, query and index tuning, backup and restore, point-in-time recovery, and HA/DR.
• Real experience with identity and access design, including service principals versus user-authorization (OBO) patterns and the tradeoffs between them.
• Practical experience building and maintaining CI/CD pipelines with GitHub Actions and/or Azure DevOps.
• Proficiency in Python or another scripting language for platform automation and tooling.
• Experience using AI tools such as GitHub Copilot, ChatGPT, or Claude to speed up engineering work.
• Comfort owning the infrastructure lifecycle, including provisioning, decommissioning, and disaster recovery.
• Experience in a regulated environment, ideally healthcare, with HIPAA and/or SOC 2 requirements.
• Comfort using elevated privileges in a regulated, PHI-bearing environment, with the judgment to escalate instead of granting exceptions on your own.
• Ability to enforce architecture and security standards owned by others, using influence and automation rather than authority.
• Strong problem-solving and troubleshooting skills, including during incidents.
• Excellent communication and collaboration skills, with the ability to work across engineering teams.
• Strong candidates will also have: exposure to data mesh or domain-oriented data architecture, and a view that platform teams exist to make other teams self-sufficient.
• Strong candidates will also have: hands-on experience with Databricks Lakebase or another cloud Postgres service (Azure Database for PostgreSQL, Neon, Aurora), and with tools such as psql, pgAdmin, and migration frameworks.
• Strong candidates will also have: enough Azure SQL Database familiarity to coordinate with the DBA team, and experience integrating OLTP systems with a lakehouse.
• Strong candidates will also have: healthcare data experience, including HIPAA-driven access controls and PHI segregation, and the ability to learn new business models and technologies.
• Strong candidates will also have: experience with observability tools such as Datadog or OpenTelemetry-based pipelines.
• Strong candidates will also have: experience with Databricks Asset Bundles, DLT, or dbt in a multi-team deployment model.
• Strong candidates will also have: relevant certifications, such as Databricks Data Engineer Professional or Azure Administrator.
• Strong candidates will also have: the ability to make logic-based decisions, focus on execution in a fast-paced setting, and recognise both small and large wins.
• Strong candidates will also have: a commitment to inclusion, value diverse perspectives, and care about the customers the work serves.
• Strong candidates will also have: drive to take on big problems, integrity in giving honest answers, and a team-first attitude.

Conditions

Hybrid: at least 3 days per week in the Dallas, TX office.

No overnight on-call rotation. Occasional availability outside core hours is expected for scheduled deployments and major incidents.

The company does not discriminate on the basis of race, sex, color, religion, age, national origin, marital status, disability, veteran status, genetic information, sexual orientation, gender identity, or any other characteristic protected by law.

Benefits

• Medical, dental, and vision insurance.
• Short- and long-term disability, and life insurance.
• 401k with company match.
• Flexible time off.
• Paid parental leave.