Senior Data Engineer (Real-Time Data Processing & Streaming)

Description

The company is hiring an experienced senior data engineer to help design, build and run scalable, high-performance data processing systems.

The work centres on real-time processing and streaming, with event-driven designs and complex pipelines that take in, transform, enrich and forward data from connected physical devices to applications and storage.

You will work alongside tech leads, software engineers and other teams in a cloud-native setting to build dependable, efficient systems that are easy to maintain.

The company has been operating for over twenty years and partners with major brands across many sectors to build platforms and digital products.

Responsibilities

• Build and maintain pipelines that process data in real time or near real time, scaling them as volumes grow.
• Build streaming solutions with Apache Kafka, Apache Flink and Apache Spark.
• Write and support Java services and integration components that handle data from physical devices.
• Write complex transformations, mappings, validations and enrichments that meet business and technical needs.
• Create and maintain event-driven links using Kafka, Kafka Streams, Kafka Connect and AMQP messaging.
• Make sure data reaches downstream consumers, applications and storage systems reliably.
• Maintain MongoDB collections, data structures and application integrations in line with architecture guidelines.
• Build, deploy and maintain cloud-native services, mainly on AWS, using Kubernetes and managed cloud services.
• Help modernise and move older data processing components onto Kafka-based streaming.
• Work with tech leads and engineering teams to turn designs into solid, production-ready code.
• Diagnose performance, reliability and data consistency problems across distributed systems.
• Improve code quality, automated testing, CI/CD practices and engineering processes over time.

Requirements

• Solid professional background in data engineering, especially streaming data and distributed systems.
• Advanced Java skills (version 11 or later), including the Spring framework.
• Hands-on Apache Kafka experience, covering Kafka Streams, Kafka Connect and building Kafka-based integrations.
• Practical use of a streaming or distributed processing framework such as Apache Flink, Apache Spark or Spark Streaming.
• Good understanding of event-driven design, asynchronous communication and messaging systems, including AMQP.
• Experience designing and implementing transformation, enrichment and integration pipelines.
• Experience building and integrating microservices and distributed applications.
• Hands-on experience with AWS services, particularly S3, SQS and EKS.
• Experience deploying and running containerised applications with Docker and Kubernetes.
• Working knowledge of MongoDB, including collections, document structures and Java-based data access.
• Familiarity with Redis and how it is used in distributed applications.
• Experience with Git-based workflows and CI/CD pipelines, ideally GitLab and GitLab CI/CD.
• Strong troubleshooting and problem-solving skills, particularly in distributed, high-throughput data environments.
• Able to work well with tech leads and cross-functional engineering teams while owning implementation and delivery.
• Experience with Apache Pulsar or Apache Storm, especially in maintaining or modernising legacy systems.
• Familiarity with Microsoft Azure and hybrid or multi-cloud environments.
• Experience processing telemetry, IoT or other event data generated by physical devices.
• Knowledge of SMTP-based integrations.
• Experience migrating legacy messaging and stream processing solutions to modern Kafka-based architectures.
• Understanding of performance optimisation, fault tolerance, monitoring and observability in distributed streaming systems.

Conditions

Hybrid working with flexible hours.

Applications from people of all backgrounds are welcome, hiring is based on merit, and adjustments are available during recruitment on request.

Benefits

• Pay: a competitive salary package, a share plan, company performance bonuses, value-based recognition awards and a referral bonus.
• Career: career coaching, global opportunities, non-linear career paths and internal programmes aimed at managers and technical leaders.
• Learning: complex projects, rotations, internal tech communities, training, certifications, coaching, online learning subscriptions, pass-it-on sessions, workshops and conferences.
• Health: a company-wide wellbeing programme, access to wellbeing apps and an employee assistance programme.
• Community: global tech communities, hobby clubs and interest groups, inclusion and diversity programmes, and events.