Senior Data Engineer (Real-Time Data Processing & Streaming)

Description

The company is hiring a senior data engineer to help design, build and run scalable, high-performance data processing systems.

The work centres on real-time processing, streaming technologies and event-driven architecture. Pipelines carry data from connected physical devices to downstream applications and storage.

You will work with tech leads, software engineers and other cross-functional teams in a cloud-native setting to build solutions that are reliable, efficient and easy to maintain.

The company is a technology consultancy that works with leading brands across industries, combining engineering skill, sector knowledge and a people-focused approach to build digital platforms and experiences. It has been operating for over two decades.

Responsibilities

• Build and maintain scalable pipelines that process data in real time or near real time.
• Develop streaming solutions using Apache Kafka, Apache Flink and Apache Spark.
• Write and maintain Java services and integration components that handle data from physical devices.
• Create complex transformations, mappings, validations and enrichments that meet business and technical needs.
• Build event-driven integrations with Kafka, Kafka Streams, Kafka Connect and AMQP messaging.
• Make sure data reaches downstream consumers, applications and storage reliably.
• Maintain MongoDB collections, data structures and application integrations in line with architecture guidelines.
• Deploy and run cloud-native services, mainly on AWS, using Kubernetes and managed cloud services.
• Help move legacy data processing components onto Kafka-based streaming architectures.
• Turn architectural designs into robust, production-ready code with tech leads and engineering teams.
• Investigate performance, reliability and data consistency problems in distributed data systems.
• Improve code quality, automated testing, CI/CD practices and engineering processes.

Requirements

• Strong professional data engineering experience, especially in streaming data and distributed systems.
• Advanced Java skills (version 11 or later), including the Spring framework.
• Hands-on Apache Kafka experience, including Kafka Streams, Kafka Connect and building Kafka-based integrations.
• Practical experience with Apache Flink, Apache Spark or Spark Streaming.
• Solid grasp of event-driven design, asynchronous communication and messaging systems, including AMQP.
• Experience designing transformation, enrichment and integration pipelines.
• Experience building and integrating microservices and distributed applications.
• Hands-on work with AWS services, particularly S3, SQS and EKS.
• Experience deploying and operating containerized applications with Docker and Kubernetes.
• Working knowledge of MongoDB, including collections, document structures and Java-based data access.
• Familiarity with Redis and how it is used in distributed applications.
• Experience with Git-based workflows and CI/CD pipelines, ideally GitLab and GitLab CI/CD.
• Strong troubleshooting skills, especially in distributed, high-throughput data environments.
• Ability to collaborate with tech leads and cross-functional engineering teams while owning implementation and delivery.
• Experience with Apache Pulsar or Apache Storm, particularly in legacy maintenance or modernization work.
• Familiarity with Microsoft Azure services and hybrid or multi-cloud environments.
• Experience processing telemetry, IoT or other event data produced by physical devices.
• Knowledge of SMTP-based integrations.
• Experience migrating legacy messaging and stream processing systems to modern Kafka-based architectures.
• Understanding of performance optimization, fault tolerance, monitoring and observability in distributed streaming systems.

Conditions

The company welcomes applicants from all backgrounds, makes hiring decisions on merit, skills, qualifications and potential, and will make adjustments for candidates who need support during recruitment.

Benefits

• Pay package with a competitive salary, a share plan, company performance bonuses, value-based recognition awards and a referral bonus.
• Career coaching, global career opportunities, non-linear career paths and internal programmes that build management and technical leadership skills.
• Learning support through complex projects, rotations, internal tech communities, training, certifications, coaching, online learning subscriptions, peer knowledge-sharing sessions, workshops and conferences.
• Hybrid working, flexible hours and an employee assistance programme.
• Company-wide wellbeing programme and access to wellbeing apps.
• Community through global tech communities, hobby clubs, interest groups, inclusion and diversity programmes, and events and celebrations.