Principal Engineer, Data Platform

Description

The company is an AI platform for market intelligence and workflow automation, used by thousands of leading organizations in business and finance.

Its content covers over 500 million premium business documents, including equity research, earnings calls, expert interviews, filings, news, and internal proprietary material.

The main platform runs on Kubernetes with a microservices design and serves most of the Fortune 500. The team is moving it to a cell-based setup, where each cell is a separate deployment.

The Data Platform role covers the tools that let engineering teams build, run, and grow their storage without each team needing deep expertise in every system. The storage stack includes Vespa, MongoDB, ClickHouse, Redis, ScyllaDB, PostgreSQL, Temporal, and Kafka, among others.

The company prefers managed services over self-hosted ones, and engineering teams own the storage they build and run. This role makes that model work at scale by providing the platform, standards, and tooling, and by keeping the platform itself reliable and scalable.

You will help set the technical direction and also write the code that builds it.

Responsibilities

• Help shape the architecture and roadmap of the Data Platform and the storage systems behind it
• Build tooling for large-scale data migrations across cells, regions, and geographies, covering moving, splitting, validating, and rolling back data with minimal downtime
• Give product teams self-service ways to provision, run, and scale their own storage
• Lead the data-layer work that cell-based architecture needs, such as data placement, tenant routing, replication, and isolation
• Set and apply the managed-first approach, using managed services where they fit and self-managed systems only when there is a clear reason
• Find and fix scalability gaps in data stores across regions, cells, and deployment models
• Create shared libraries that let product teams use storage capabilities on their own
• Work closely with the Cloud Platform, SRE, and DevEx teams
• Raise the architecture standard across the organization and mentor Staff and Senior engineers
• Use AI tools for data platform tasks such as reviewing slow queries and diagnosing incidents

Requirements

• Deep hands-on experience designing and running data platforms and storage systems
• Production experience at large scale with stores such as MongoDB, ClickHouse, Redis, or PostgreSQL
• Cloud-native data storage experience, including managed data services on AWS
• Strong background in Kubernetes and containers
• Track record of designing and running large-scale data migrations, including live migrations
• Experience with cell-based, multi-tenant, or multi-region architectures
• Experience building platforms and tools that other engineers use, and being comfortable with engineers owning and running their own storage
• Proven ability to land significant changes in code owned by other teams
• A record of catching data scalability and reliability limits before they cause incidents
• Ability and drive to design, lead, influence, and deliver on ambitious technical roadmaps
• Experience running storage systems on top of Kubernetes
• Experience building control planes with Crossplane
• Experience with private, isolated, on-premises, or air-gapped deployments

Conditions

Base pay is set from standard ranges by function and level, benchmarked against similar growth-stage companies and internal peers

Base compensation range is $246,000 - $339,000 USD

Final offers depend on factors including candidate experience and expertise, so they may fall outside the range

Salary ranges are shown on all postings, whatever the hiring location

The company is an equal-opportunity employer and does not discriminate on any basis protected by law.

Benefits

• Performance-based bonus, which you may be offered
• Equity, which you may be offered
• A generous benefits program, which you may be offered