You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Citi

Lead Data Engineer – Spark / Scala - Vice President

  • Hybrid
  • 6+ years

Salary

Not stated

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The team is a data engineering group focused on reconciliation, building and running large-scale platforms that keep critical financial data accurate, consistent and complete. It is an established team with a solid foundation and a proven record, and it is now moving into a modernisation phase.

The platform is a large Scala Spark reconciliation system that is being moved from a legacy Cloudera Hadoop environment to Databricks on AWS, with cloud-native patterns, simpler pipelines and new capabilities. The work is technically demanding and directly affects data quality across the organisation.

The team sits between data engineering and financial data operations. Its processing, validation and reconciliation of large data volumes feeds downstream reporting, risk and decision-making, so platform reliability matters.

The team works daily with senior architects and platform teams on technical direction, with DevOps and cloud engineers on infrastructure, deployments and AWS adoption, and with business and stakeholder teams to turn reconciliation requirements into scalable engineering solutions.

This is a hands-on role in a global, cross-functional setting. You will help shape architecture decisions as well as implement them.

Responsibilities

  • Build and optimise large-scale Scala Spark pipelines on Databricks for reconciliation, writing clean, modular, production-ready code.
  • Refactor legacy Hadoop pipelines into cloud-native Databricks and AWS architectures using Delta Lake and Databricks Workflows.
  • Analyse performance bottlenecks in complex or over-engineered Spark jobs and simplify them into efficient, standard patterns.
  • Turn high-level architecture decisions into detailed technical designs, data models and reusable pipeline components.
  • Take part in daily design discussions, technical decisions and production releases with architects, platform and DevOps teams.
  • Write and maintain unit, integration and data validation tests that keep reconciliation pipelines reliable.
  • Contribute to shared frameworks, reusable libraries and best practices across the data platform.
  • Help drive the migration from legacy Hadoop to Databricks on AWS while steadily improving the quality and efficiency of reconciliation processing at scale.
  • Work with regional and local business and data operations colleagues who rely on reconciliation outputs.

Requirements

  • Experience with Scala Spark, PySpark or Java Spark, and comfort moving between Spark language APIs.
  • Familiarity with AWS services such as S3, Glue, IAM, CloudWatch or Step Functions in data engineering work.
  • Exposure to data reconciliation or financial data, including reconciliation logic, data validation and accuracy requirements at scale.
  • Experience with CI/CD pipelines and DevOps practices, including automated testing, deployment pipelines and version control for data engineering.
  • Knowledge of data modelling and of pipeline orchestration beyond Databricks Workflows, such as Apache Airflow.
  • Prior work on large-scale Hadoop to cloud migrations or legacy platform modernisation.
  • Familiarity with streaming data processing, such as Apache Kafka or Spark Structured Streaming, alongside batch workloads.
  • Experience contributing to shared engineering frameworks or reusable libraries within a platform team.

Conditions

Full-time hybrid role with up to 2 days working from home per week, in a business casual workplace.

Base salary is competitive and reviewed annually.

Equal opportunity employer; the company welcomes people to bring their whole selves to work.

Benefits

  • 27 days annual leave plus bank holidays.
  • Discretionary annual performance-related bonus.
  • Private medical care and life insurance.
  • Employee assistance programme.
  • Pension plan.
  • Paid parental leave.
  • Discounts for employees, family and friends.
  • Access to a range of learning and development resources.

Where you’d work

Part of the week in the office

You can work from

  • United Kingdom

About the company

Citi

  • Industry: FinTech

Office in Belfast, United Kingdom

Also hiring in Budapest, Hungary, Canada, Jersey City, United States and 13 more places

Your chances

Worth a look before you spend an evening tailoring a CV for it.

  • 19 checks run
  • 1 red flag

Still hiring?

13 checks

1 red flag

How crowded?

6 checks

No red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details13 facts · Role, Location, Compensation, Employment, Company
Tech stack
  • AWS
  • Kafka
  • Spark
  • Databricks
  • Airflow
Seniority
VP / C-level
Type
Full-time
Industry
FinTech
Specialty
Data Engineering
Region
United Kingdom
Show 7 more factsShow less

Role

Category
Data & Analytics
Specialty
Data Engineering
Seniority
VP / C-level
Experience
6+ years
Tech stack
  • AWS
  • Kafka
  • Spark
  • Databricks
  • Airflow

Location

Work model
Hybrid
Region
United Kingdom
Office
  • Belfast, United Kingdom
Remote from
  • United Kingdom

Compensation

Salary
Salary by agreement
Pay period
Annual

Employment

Type
Full-time

Company

Industry
FinTech

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason