The DGX Cloud organization is hiring a principal expert to lead the design, build, and running of the DGXC Data Platform. The group connects data from cloud service providers, the company Cloud Partners, and internal systems, and provides shared data functions so DGXC teams can publish, find, join, and use dependable data in their operational workflows.
You will turn complicated operational needs into reliable platform services and shared datasets. You will set the technical direction for ingestion and orchestration, data agreements, quality monitoring, cataloging, and controlled access, drive adoption across teams, define production standards, and resolve technical problems that span several organizations.
The aim is less duplicated integration work, more trust in operational data, and a platform that is simpler to adopt and run. The main stack is Python, SQL, Databricks, and Spark.
Responsibilities
Set the architecture and a multiyear technical roadmap, including system boundaries, shared interfaces, and engineering standards for ingestion, processing, storage, and serving. Weigh near-term operational needs against long-term reliability, maintainability, and cost.
Direct delivery across platform engineers, data producers, and data consumers. Turn unclear requirements into concrete designs and milestones, clear blockers, and carry work through to production adoption.
Build and run reusable batch and streaming ingestion for service provider and internal data. Create supported patterns for APIs, change data capture, streams, database connectors, and files, covering orchestration, retries, backfills, and recovery.
Define shared identifiers, data contracts, consistent models, and automated quality and certification. Work with source owners to settle what each dataset means, its keys, valid joins, historical coverage, compatibility rules, and ownership. Make quality failures easy to act on and make dataset limits visible to consumers.
Lead the work on cataloging, lineage, and governed access so that people and AI agents can find datasets, understand their definitions and relationships, and follow approved access steps.
Make onboarding and operations repeatable with shared libraries, reference implementations, deployment automation, and self-service workflows that teams can use in their own repositories. Track onboarding effort, dataset reuse, and manual toil over time.
Own production readiness and reliability standards: SLOs, testing, observability, runbooks, rollback, and recovery. Lead major investigations across pipelines, applications, SQL engines, Spark, storage, networks, and cloud services. Work with security and infrastructure teams on service identities, least privilege, secrets, environment isolation, auditability, and upgrades or migrations that keep data access and production running.
Mentor senior engineers and technical leads, run design reviews, and settle technical disagreements using evidence and explicit tradeoffs. Work with leadership on priorities, progress, operational risk, and investment decisions.
Requirements
Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent experience.
More than 16 years of steadily growing experience building and running production software, data platforms, databases, or distributed systems.
A track record of setting technical strategy and architecture for shared platforms used by several teams, and being accountable for delivery, adoption, and production results. You show sound judgment on reliability, performance, security, cost, and maintainability by testing and comparing options.
Extensive hands-on work with distributed data processing, analytical databases, production ETL, streaming systems, or backend platforms handling large data volumes. You understand how failures spread across compute, storage, networking, and cloud services.
Strong software engineering fundamentals and production-level Python or another backend or systems language. You are able and willing to work mainly in Python and SQL, have designed reusable frameworks, and have written and debugged critical code yourself.
Deep SQL and data modeling skills, including query execution, incremental processing, schema evolution, and consistency. You have linked independently run systems through shared identifiers, documented relationships, and reliable data contracts with clear ownership. You can reason about idempotency, replay, late-arriving data, partial failure, and correctness across system boundaries.
Experience making production practices that several teams adopt, such as CI/CD, data validation, observability, incident response, secure deployment, and recovery. You have led complex investigations and major migrations, using logs, metrics, traces, query plans, profiles, and controlled experiments to find root causes and prevent repeats.
A proven ability to shape technical direction across organizational lines, get engineering and business leaders aligned on shared goals, and grow engineers into broader technical roles.
Conditions
Applications will be accepted at least until November 13, 2026.
This is an existing vacancy.
The company uses AI tools in its hiring process.
The company is an equal opportunity employer and does not discriminate on protected characteristics.
Benefits
The company describes its salaries and benefits as highly competitive and comprehensive. Details are available on the company's benefits site.