Principal Data Architect

Description

The company provides AI and data analytics for identity intelligence and risk operations, serving government and commercial clients. Its platform delivers analysis-ready data in any language, flags risk early, gives 360-degree views, automates at high speed and integrates with existing systems.

This is a hands-on principal architect role to redesign the company's data infrastructure and pipelines. You will set the target architecture, write key code yourself and lead engineers through a multi-year move of a petabyte-scale platform to Google Cloud.

Data is collected, enriched and resolved from hundreds of open-source and commercial sources in dozens of languages. It currently sits in separately built pipelines and stores, including a mix of warehouse, search and object storage plus legacy delivery systems that are hard to change and costly to run. The goal is one platform that is cheaper, more reliable and ready for AI and agentic workloads.

This is an individual contributor role, not a people-management one. You will be measured on the systems you design and ship, the quality of your technical decisions and how much you lift the engineers around you. You report to the SVP of Engineering and work with the Head of Architecture and the AI, Graph and Product teams.

Responsibilities

• Within 90 days, map the whole current estate: each source, pipeline, store, consumer and delivery path, with its volume, cost, owner and failure history. Produce a target data architecture and a sequenced migration plan that engineering and product leaders approve.
• Build a standard backbone for streaming and batch ingestion, enrichment and delivery to replace custom pipelines. It must handle millions of documents a day against clear SLOs.
• Design ingestion and processing for idempotency, backfill, replay, schema evolution and, where it matters, exactly-once or effectively-once delivery.
• Lead the data platform part of the GCP migration. For each wave, decide what moves unchanged, what is rebuilt and what is retired, and own that wave's technical design.
• Define when to use the warehouse (BigQuery), search and indexing (Elasticsearch/OpenSearch), object storage, vector stores and graph stores. Consolidate toward that pattern and cut the cost of the most expensive workloads.
• Own the physical platform across warehouse, lakehouse, search and object storage, including partitioning, indexing, tiering and retention at petabyte scale, and the hardware and cost trade-offs behind query performance.
• Design the move from AWS to GCP with minimal customer disruption, including patterns that also work in FedRAMP, disconnected and air-gapped deployments without managed cloud services.
• Build the data foundation for AI: agent-callable data services, hybrid retrieval across keyword, analytical, vector and graph data, and provenance and lineage on every record so AI outputs can cite their evidence.
• Build APIs and query layers for analysts and agents, working with the AI and Graph teams on RAG and agentic workflows.
• Build a governance and compliance platform that centralizes and serves governed, compliant data, including under privacy laws. Set the parameters, exclusions and retention rules that get the best AI performance.
• Make the platform operable with data contracts, automated quality checks, observability and cost attribution, so every pipeline has an owner, an SLO and a known unit cost.
• Track unit economics for cost per source, per pipeline, per query and per customer.
• Write architecture decision records, run design reviews and set coding standards for data engineering.
• Mentor senior engineers, partner with the offshore engineering teams and act as the escalation point for the hardest data incidents.

Requirements

• 12+ years building data systems, at least 5 of them as lead architect or principal engineer on a large-scale data platform.
• You still write production code: strong Python plus at least one of Java, Scala or Go, and fluent SQL.
• You have personally led a major re-architecture or cloud migration of a data platform and can explain the design, the sequencing, what broke and what you would change.
• Deep hands-on experience with distributed processing and streaming, such as Spark, Beam/Dataflow, Flink, Kafka, Pub/Sub or Pulsar.
• Production experience with GCP data services such as BigQuery, Dataflow, Pub/Sub, GCS and Spanner. AWS experience is also expected, since you will be migrating off it.
• Expert at running search and index systems (Elasticsearch or OpenSearch) at scale, including cluster sizing, mappings, aggregation performance and cost tuning.
• Experience with lakehouse table formats (Iceberg, Delta or Hudi) and orchestration (Airflow/Composer, Dagster or similar).
• A record of cutting platform cost materially without hurting performance, with the numbers to show it.
• Clear written communication: you can write a design doc that both a CEO and a senior engineer understand.
• Ability to obtain a US security clearance.
• Bachelor's degree in Computer Science, Engineering or a related technical field, or equivalent hands-on experience.
• Preferred: experience with vector search, embedding pipelines and retrieval architectures for LLM and agentic applications.
• Preferred: experience with entity resolution, knowledge graphs or graph databases, ideally on multilingual or messy real-world data.
• Preferred: background in OSINT, national security or law enforcement, risk intelligence or financial data.
• Preferred: experience building for FedRAMP, IL5 or air-gapped environments, including running data platforms without managed cloud services.
• Preferred: experience with data observability, lineage and governance tools such as OpenLineage, Great Expectations, Monte Carlo or Dataplex.
• Preferred: experience combining large volumes of scraped multimedia with structured third-party data.
• A Master's or PhD is a plus but not required.

Conditions

Base pay range is $220,000 - $250,000 USD, set by qualifications and experience.

The company is headquartered in Reston, Virginia, with regional offices in Boston, MA and Cleveland, OH, and international offices in Australia, Canada, Israel, Japan and the U.K.

Benefits

• Medical, dental, vision, life and disability insurance, with the company covering 85-100% of monthly premiums for you and your family.
• Traditional and Roth 401(k) with a very competitive match.
• Unlimited flexible leave to manage your own time and balance work and personal life.
• 12 paid Federal Holidays.
• Tuition reimbursement for continuing education.