Senior Data Engineer (Real-Time Data Processing & Streaming)

Description

The company is a long-established consultancy of more than 20 years that partners with well-known brands across industries to build digital platforms and intelligent experiences.

The role centres on real-time data processing and streaming, including event-driven architectures. You will work on pipelines that take in data from connected physical devices, process it, and pass it on to downstream applications and storage.

You will work alongside Tech Leads, software engineers and other cross-functional teams to build dependable, efficient, maintainable data processing systems in a cloud-native setting.

Responsibilities

• Build and maintain pipelines that process data in real time or near real time, and that scale well.
• Create streaming solutions with Apache Kafka, Apache Flink and Apache Spark.
• Write and maintain Java services and integration components that handle data coming from physical devices.
• Apply complex transformations, mappings, validations and enrichments that meet business and technical needs.
• Build and run event-driven integrations with Kafka, Kafka Streams, Kafka Connect and messaging protocols such as AMQP.
• Make sure data reaches downstream consumers, applications and storage systems reliably.
• Set up and maintain MongoDB collections, data structures and the application-level integrations around them, following architecture guidelines.
• Develop, deploy and maintain cloud-native services, mainly on AWS, using Kubernetes and managed cloud services.
• Help modernise and migrate legacy data processing components to Kafka-based streaming architectures.
• Turn architectural designs into robust, production-ready code with technical leads and engineering teams.
• Investigate performance, reliability and data consistency problems across distributed data systems.
• Help improve code quality, automated testing, CI/CD practices and engineering processes.

Requirements

• Solid professional experience in data engineering, with a focus on streaming data and distributed systems.
• Advanced Java skills (Java 11+), including working with the Spring framework.
• Hands-on work with Apache Kafka, including Kafka Streams, Kafka Connect and building Kafka-based integrations.
• Practical experience with streaming or distributed processing frameworks such as Apache Flink, Apache Spark or Spark Streaming.
• A good grasp of event-driven architectures, asynchronous communication patterns and messaging systems, including AMQP.
• Experience designing and building data transformation, enrichment and integration pipelines.
• Experience building and integrating microservices and distributed applications.
• Hands-on use of AWS services, particularly S3, SQS and EKS.
• Experience deploying and running containerised applications with Docker and Kubernetes.
• Working knowledge of MongoDB, including collections, document structures and Java data access.
• Familiarity with Redis and how it is used in distributed applications.
• Experience with Git-based workflows and CI/CD pipelines, ideally GitLab and GitLab CI/CD.
• Strong troubleshooting and problem-solving skills, especially in distributed, high-throughput data environments.
• Ability to work well with technical leads and cross-functional engineering teams, while owning implementation and delivery.
• Experience with Apache Pulsar or Apache Storm, particularly in maintaining or modernising legacy systems.
• Familiarity with Microsoft Azure services and hybrid or multi-cloud setups.
• Experience handling telemetry, IoT or other event data produced by physical devices.
• Knowledge of SMTP-based integrations.
• Experience moving legacy messaging and stream processing solutions to modern Kafka-based architectures.
• Understanding of performance tuning, fault tolerance, monitoring and observability in distributed streaming systems.

Conditions

The role is hybrid, with flexible working hours.

Equal opportunity applies, and hiring is based on merit, skills, qualifications and potential. Support or adjustments during recruitment are available on request.

Benefits

• Competitive salary package, share plan, company performance bonuses, value-based recognition awards and a referral bonus.
• Career coaching, global career opportunities, non-linear career paths and internal development programmes for management and technical leadership.
• Complex projects, rotations, internal tech communities, training, certifications, coaching, online learning platform subscriptions, pass-it-on sessions, workshops and conferences.
• Employee assistance programme, a global internal wellbeing programme and access to wellbeing apps.
• Global internal tech communities, hobby clubs and interest groups, inclusion and diversity programmes, and events and celebrations.