Senior Data Platform Engineer

Description

The company's aim is to reduce the global burden of disease, and it believes better use of data supports that aim.

The company built the Healthcare Map, a view of the U.S. healthcare system that combines de-identified, real-world patient data with algorithms and clinical experience. It underpins a set of software applications that help partners answer complex healthcare questions, such as tracking patient behavior and treatment patterns, finding gaps in care, and meeting unmet patient needs.

You will join the Data Foundations team, which builds the core data products behind the Healthcare Map. The work turns very large, complex healthcare datasets into fast, reliable data assets for customer products, internal projects, and AI and ML use cases.

The role is a hands-on engineering position focused on large-scale pipelines and the data platform. You will work on processing complex healthcare data at scale, improving reliability and speed, and shaping the technical direction of core systems.

Within your first 12 months, you are expected to deliver technical work that improves pipeline performance, scalability, and efficiency; make core systems more reliable, observable, and cost-effective; build healthcare domain knowledge and help develop methods for patient journey mapping and identity resolution; ship production-grade data solutions with Data Product and Engineering; and raise the team's standards through mentoring, design reviews, and engineering practice.

The company is an ambitious, supportive group with varied backgrounds, and it values growth, strong work, and enjoying the journey.

Responsibilities

• Build, run, and tune large batch pipelines on AWS using Python, SQL, Spark, and Airflow.
• Decide how data is stored and laid out, including table formats, partitioning, file sizes, and compaction, so large datasets stay quick to query and affordable.
• Ingest data from many external sources and keep pipelines accurate as schemas change often.
• Turn healthcare claims, EHR, and reference data into trusted data products that customers and internal teams can use directly.
• Improve reliability with data quality checks, validation, lineage, monitoring, and alerts.
• Investigate complex data, system, and performance problems in compute-heavy workflows.
• Help with system design, code quality, testing, documentation, CI/CD, and rotating production support.

Requirements

• Proven experience building, owning, and debugging production data pipelines at multi-terabyte scale.
• Advanced Python for memory-efficient, high-performance data processing, plus strong SQL.
• Deep production experience with Airflow, including running and scaling it, not only writing DAGs.
• A solid grasp of physical data storage, including columnar and row formats and open table formats such as Iceberg, Delta Lake, or Hudi, with attention to updates, compaction, small files, and partitioning.
• Experience with Spark or a similar distributed processing framework.
• Proven experience designing and running data systems on AWS.
• Strong judgment on data correctness, root-cause analysis, and production troubleshooting.
• Ability to explain technical trade-offs clearly to engineering, product, and data partners.
• Comfort using AI-assisted engineering tools for coding, debugging, documentation, and technical research.
• You are expected to use AI engineering tools such as ChatGPT, Gemini, or Claude to work faster and make better technical decisions. This includes generating and refining code, speeding up documentation, creating test cases, debugging hard issues, learning unfamiliar concepts, and weighing architectural trade-offs and risks.
• Nice to have: experience with containers and orchestration such as Docker and Kubernetes.
• Nice to have: strong software engineering fundamentals and experience with more than one language.
• Nice to have: data governance and lineage experience.
• Nice to have: experience delivering external data products through APIs, serving layers, or customer access patterns.
• Nice to have: experience applying AI or agentic workflows to engineering, data quality, delivery, or operations.
• Nice to have: experience working in high-growth or ambiguous environments.
• No healthcare background is required.

Conditions

Hybrid model with hubs in San Francisco, New York City, and Chicago. Some roles can be done from anywhere in the country, some are tied to a specific region, and some are based near a hub. Hub-based staff follow planned in-office schedules alongside flexible working. Expectations are made clear before you join.

Open to remote work in the US, or hybrid in San Francisco or New York City.

Starting annual base pay is $196,000 to $230,000 USD in the San Francisco Bay Area and New York City, and $170,000 to $200,000 USD in all other US locations. Pay is set using skills, experience, location, training, business needs, and market demand.

The company uses AI as a core part of its work and expects every team member to bring AI into daily tasks, from summarizing documents to automating workflows and finding insights.

Benefits

• Performance-based bonus, at the company's sole discretion and under a written agreement or plan.
• Possible equity awards.
• Comprehensive health, dental, and vision insurance.
• Flexible time off and holidays.
• 401(k) with company match.
• Disability and life insurance.
• Leaves of absence in line with applicable state and local laws and company policy.