You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Cloudflare

Senior Data Engineer, Cloudforce One

  • Hybrid
  • 6+ years

Salary

Not stated

Similar roles pay $145K - 210K a year · our estimate

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The team sits in Cloudforce One, the threat operations and research group that finds and disrupts cyber threats, from organised cybercrime to nation-state advanced persistent threats. It works with data from one of the largest global networks, analysing it at scale and turning the results into threat intelligence that protects customers.

You will join the data science team, which powers insight behind the Threat Events product. Customers use that product to protect themselves, for example by turning threat indicators into real-time WAF rules. The team owns pipelines feeding analysis of DDoS and WAF data, including the quarterly DDoS Threat Report, and supports Trust & Safety work on network abuse. It also works closely with the Threat Intelligence team.

The role is about building the data foundation for this work: scalable pipelines, analytical stores and data marts that turn raw telemetry into trusted, well-modelled datasets for researchers, data scientists, analysts and customers.

The role may involve on-call work outside standard hours to resolve technical issues.

The company runs a large global network that protects and speeds up internet applications, and the company was named to Entrepreneur's Top Company Cultures list and Fast Company's World's Most Innovative Companies list.

Responsibilities

  • Build and run batch pipelines on Kubernetes-hosted Airflow and Spark that extract, enrich and publish threat and abuse signals from network and platform telemetry.
  • Develop and evolve Apache Iceberg (R2) data stores and the curated marts used by Threat Events, Trust & Safety and Threat Intelligence investigations.
  • Design datasets deliberately, choosing grain, partitioning and schema evolution so they stay correct, queryable and cost-efficient.
  • Write and tune SQL on large analytical data in ClickHouse, balancing completeness, speed and cost.
  • Make pipelines safe to retry, resume and backfill, so consumers get exactly the data intended for publication.
  • Build data quality checks, reconciliation and monitoring that catch problems before analysts, detections or customers see them.
  • Treat privacy and data minimisation as requirements from the first design stage.
  • Give Threat Intelligence and Trust & Safety partners the data they need, turning one-off requests into reusable datasets.
  • Work with data scientists to turn exploratory analysis into repeatable production data products and ML-ready datasets.
  • Raise the engineering standard through code review, testing, CI/CD and documentation.

Requirements

  • 7+ years of experience building and running scalable production data pipelines.
  • Proven experience building analytical data stores and data marts, including modelling, partitioning and schema evolution.
  • Strong Python for data engineering, with tests and well-structured, maintainable code.
  • Strong SQL, including window functions, query optimisation and tuning on large datasets.
  • Hands-on production experience with Apache Airflow, including Kubernetes-hosted deployments.
  • Experience with distributed processing at large scale, such as Spark or similar.
  • Experience with Apache Iceberg or a comparable open table format on object storage such as R2 or S3.
  • Experience with ClickHouse or a comparable columnar analytical database.
  • Experience with CI/CD, version control and containerised deployments.
  • Clear communication, and the ability to turn unclear requirements from researchers and analysts into well-defined data products.
  • Bonus: experience with security, threat intelligence, fraud, or trust and safety or abuse data.
  • Bonus: a solid understanding of how the internet works, including DNS, HTTP, TLS and BGP.
  • Bonus: experience supporting data science or ML workflows, such as feature stores and training datasets.
  • Bonus: experience with the company's developer platform, including Workers, R2 and R2 Data Catalog.
  • Bonus: experience with Trino, Presto or other distributed query engines.
  • Bonus: experience using AI-assisted development tools in your workflow.

Conditions

Location: London, UK.

Possible on-call duties outside standard working hours.

Applicants who reach the offer stage may be asked to attend an in-person interview at a the company office or hub.

The role may require access to information controlled under US export laws, including the Export Administration Regulations. Any offer may be conditional on your authorisation to receive such software or technology without sponsorship for an export licence.

Where you’d work

Part of the week in the office

No visa sponsorship

About the company

Cloudflare

  • Industry: Cybersecurity

6 of their 98 open roles are remote

Your chances

Worth a look before you spend an evening tailoring a CV for it.

  • 19 checks run
  • 1 red flag

Still hiring?

13 checks

No red flags

How crowded?

6 checks

1 red flag

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details11 facts · Role, Location, Compensation, Company
Tech stack
  • Python
  • SQL
  • Kubernetes
  • Spark
  • Airflow
Seniority
Senior
Industry
Cybersecurity
Specialty
Data Engineering
Region
United States
Pay period
Annual
Show 5 more factsShow less

Role

Category
Data & Analytics
Specialty
Data Engineering
Seniority
Senior
Experience
7+ years
Tech stack
  • Python
  • SQL
  • Kubernetes
  • Spark
  • Airflow

Location

Work model
Hybrid
Region
United States
Visa sponsorship
Not sponsored

Compensation

Salary
Salary by agreement
Pay period
Annual

Company

Industry
Cybersecurity

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason