You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Early Window: Closes in
Company hidden

Principal Engineer, Data Platform

  • Office
  • 6+ years

Salary

$275,000 - 350,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company builds an AI Site Reliability Engineer for large enterprises. Its agent autonomously reads and reasons across very large production environments to diagnose, fix and prevent incidents, which matters more as AI-generated code adds to production complexity.

The company already runs in mission-critical settings at large enterprises such as PepsiCo, Nubank, Harvey and DigitalOcean. Its roots are in AI research, and its team combines researchers from MIT, Harvard, Berkeley, Columbia and Cornell with engineers and operators from Google, Meta, Datadog, ServiceNow and Citadel Securities.

The role is a Principal Engineer position on the Data Platform team, based in New York.

Responsibilities

  • Own the technical direction of the data platform, which is the foundation the agent's investigations depend on. It is a petabyte-scale system that takes hundreds of billions of log lines a day from one customer, at hundreds of megabytes per second, and makes them queryable within minutes.
  • Set the long-term architecture for four data paths: direct push ingestion, scheduled pulls from third-party observability platforms, non-telemetry data, and transformation pipelines. Specify the interfaces, invariants and evolution path so they scale together.
  • Choose the storage and query design for 10-100x today's scale, weighing time-series, columnar/OLAP and lakehouse options, along with retention, cost and performance targets for the agent's key queries.
  • Treat the agent as a core workload. Set performance targets for its tool queries, freshness guarantees for its data, and safe sandboxing for the ad-hoc queries an LLM will write.
  • Work with the AI Research team to supply investigation history, evaluation datasets and derived signals for the agent and its auto-research loops.
  • Build the orchestration and transformation layer: scheduled, idempotent, observable jobs that bring in non-telemetry context and produce materialized views, rollups, entity graphs, and LLM-based summaries and enrichment cached before investigations begin.
  • Define and meet SLOs for availability, data freshness and query latency. Ship systems with monitoring, runbooks and safe rollouts, and lead the technical response when failures occur.
  • Design redaction, tenant isolation and data residency controls that meet the security and compliance standards of large banks, payment companies and tech firms.
  • Mentor senior engineers and set standards and reference implementations that other teams adopt.
  • Stay hands-on: write the design docs, write the code that tests them, and join architecture reviews and Fortune 100 customer deployments. Bring what you learn from customers and go-to-market teams back into the roadmap.
  • Work directly with engineering and company leadership on the roadmap and its business impact.

Requirements

  • Deep distributed-systems knowledge across streaming data, analytical storage and query engines, such as Kafka or equivalent, ClickHouse, Doris, Druid or Pinot, and lakehouse technologies, plus experience building reliable multi-tenant systems.
  • Strong systems programming in Rust, the platform's main language, or comparable Go experience, and comfort with Python.
  • Experience deploying on Kubernetes, Helm, Terraform and AWS, ideally in environments you do not fully control, such as BYOC, on-prem or regulated customer clouds.
  • A record of leading multi-quarter migrations through adoption and deprecation, not just design.
  • You write production code regularly and can walk through recent code you personally shipped.
  • Sound cost judgment, weighing performance, cost, durability and simplicity with real numbers, and knowing what your systems cost to run.
  • Clear technical writing, the ability to make firm tradeoff decisions, build alignment without formal authority, and hold your own with enterprise customers in architecture reviews.
  • Experience building the backend of an observability, logging, APM or SIEM product.
  • Fluency with OpenTelemetry and OTLP, and with the data models and APIs of major observability vendors such as Datadog, Splunk, Elastic, Prometheus and Dynatrace.
  • Experience with workflow orchestration and pipeline tools such as Airflow, Dagster or Temporal.
  • Experience building LLM-powered pipelines for enrichment, summarization and embeddings, and controlling their cost.
  • Background in data governance, including PII redaction, data residency and regulated industries such as financial services.
  • Contributions to open-source data infrastructure.

Conditions

Full-time, in-person role in New York, near Madison Square Park, five days a week.

The U.S. base salary range is $275,000 to $350,000 USD, plus equity and benefits. Pay depends on location, level and role, and individual offers reflect experience, skills and job-related knowledge.

The company is fully in-office, with a collaborative culture focused on AI-powered software maintenance.

Benefits

  • Startup equity, health insurance, fertility benefits, a technology setup stipend and flexible time off.
  • In-office snacks, team happy hours and outings, an annual company offsite, and built-in time to collaborate across teams.

Where you’d work

From the office, 4 or more days a week in the office

About the company

Company hidden

  • Industry: AI

Office in New York, United States

Your chances

Still hiring, not crowded yet, and you'd be among the first.

  • 18 checks run
  • 9 good signs
  • 1 red flag

Still hiring?

11 checks

Actively hiring

In its favour4

  • Still on the company's own careers site, checked 1 h agoModerate evidence
  • Found in the last 48 hours, before the big job boardsModerate evidence
  • Specific about the basics: pay, place, level, stack and contract all statedSlight evidence
1 moreFewer
  • States its salarySlight evidence

How crowded?

7 checks

Low

In its favour5

  • In its Early Window: not on the big job boards yetStrong evidence
  • Asks for 10+ years: a narrow poolModerate evidence
  • In the office in New York: only people nearby can take itSlight evidence
2 moreFewer
  • In the office 4 or more days a week: most applicants want lessSlight evidence
  • Asks for Airflow, which only 1% of open roles doSlight evidence

Against it1

  • Pays more than 95% of similar roles: that draws applicantsSlight evidence

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details14 facts · Role, Location, Compensation, Employment, Company
Tech stack
  • Python
  • AWS
  • Kubernetes
  • Kafka
  • Airflow
  • LLMs
Seniority
Principal
Type
Full-time
Equity
Equity offered
Industry
AI
Specialty
Data Engineering
Show 8 more factsShow less

Role

Category
Data & Analytics
Specialty
Data Engineering
Seniority
Principal
Experience
10+ years
Tech stack
  • Python
  • AWS
  • Kubernetes
  • Kafka
  • Airflow
  • LLMs

Location

Work model
Office
Region
United States
Office
  • New York, United States
Days in the office
4 or more days

Compensation

Salary
$275,000 - 350,000 / year
Pay period
Annual
Equity
Equity offered

Employment

Type
Full-time

Company

Industry
AI

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason