Senior Data Engineer (Real-Time Data Processing & Streaming)

Description

You will help plan, build and run data processing systems that must scale and perform well.

The work centres on real-time processing, streaming technology and event-driven design, covering pipelines that take in, transform, enrich and send data from connected physical devices to downstream applications and storage.

You will work day to day with tech leads, software engineers and other cross-functional teams to deliver dependable, efficient and maintainable data solutions on cloud-native infrastructure.

The company consults for major brands across many industries and has been doing so for over 20 years, building digital platforms and experiences for them.

Responsibilities

• Build and maintain pipelines that process data in real time and near real time.
• Use Apache Kafka, Apache Flink and Apache Spark to implement streaming data processing.
• Write and support Java services and integration components that handle data from physical devices.
• Create complex transformations, mappings, validation and enrichment logic that meets business and technical needs.
• Build and run event-driven integrations with Kafka, Kafka Streams, Kafka Connect and AMQP messaging.
• Make sure data reaches consumers, applications and storage systems reliably.
• Maintain MongoDB collections, data structures and application integrations in line with architecture guidelines.
• Develop, deploy and run cloud-native services mainly on AWS, using Kubernetes and managed cloud services.
• Help move legacy data processing components onto Kafka-based streaming.
• Turn architecture designs into robust, production-ready code alongside technical leads and engineers.
• Investigate performance, reliability and data consistency problems across distributed systems.
• Improve code quality, automated testing, CI/CD and engineering processes.

Requirements

• Solid professional background in data engineering, focused on streaming data and distributed systems.
• Advanced Java skills (version 11 or later), with experience in the Spring framework.
• Hands-on Apache Kafka work, including Kafka Streams, Kafka Connect and Kafka-based integrations.
• Practical use of Apache Flink, Apache Spark or Spark Streaming.
• Good grasp of event-driven design, asynchronous communication patterns and messaging systems, including AMQP.
• Experience designing and building data transformation, enrichment and integration pipelines.
• Experience building and integrating microservices and distributed applications.
• Hands-on AWS experience, particularly with S3, SQS and EKS.
• Experience deploying and running containerised applications with Docker and Kubernetes.
• Working knowledge of MongoDB, including collections, document structures and Java-based data access.
• Familiarity with Redis and how it is used in distributed applications.
• Experience with Git-based workflows and CI/CD pipelines, ideally GitLab and GitLab CI/CD.
• Strong troubleshooting skills, especially in distributed, high-throughput data environments.
• Able to work with technical leads and cross-functional engineering teams while owning implementation and delivery.
• Experience with Apache Pulsar or Apache Storm, especially in legacy system maintenance or modernisation.
• Familiarity with Microsoft Azure services and hybrid or multi-cloud environments.
• Experience processing telemetry, IoT or other event data generated by physical devices.
• Knowledge of SMTP-based integrations.
• Experience migrating legacy messaging and stream processing solutions to Kafka-based architectures.
• Understanding of performance optimisation, fault tolerance, monitoring and observability in distributed streaming systems.

Conditions

Hybrid working with flexible working hours.

Applicants are assessed on merit, skills, qualifications and potential, and reasonable adjustments can be arranged during recruitment on request.

Benefits

• Competitive salary package and share plan.
• Company performance bonuses, value-based recognition awards and a referral bonus.
• Career coaching, global career opportunities, non-linear career paths and internal programmes for management and technical leadership.
• Complex projects, rotations, internal tech communities, training, certifications, coaching, online learning subscriptions, pass-it-on sessions, workshops and conferences.
• Employee assistance programme, a global wellbeing programme and access to wellbeing apps.
• Hobby clubs, interest groups, inclusion and diversity programmes, and company events and celebrations.