Senior Data Engineer (Real-Time Data Processing & Streaming)

Description

The company is hiring an experienced senior data engineer to help design, build and run scalable, high-performance data processing systems.

The work centres on real-time processing and streaming, plus event-driven designs, with pipelines that take in, transform, enrich and send on data from connected physical devices to downstream apps and storage.

You will work alongside tech leads, software engineers and other teams in a cloud-native setting. The company has been in technology consulting for more than 20 years and builds platforms and digital products for large brands in many industries.

Responsibilities

• Build and upkeep pipelines that process data in real time or near real time.
• Put streaming solutions in place with Apache Kafka, Apache Flink and Apache Spark.
• Write and maintain Java services and integration code that handles data coming from physical devices.
• Create complex transformations, mappings, validation rules and enrichments that meet business and technical needs.
• Develop and maintain event-driven integrations using Kafka, Kafka Streams, Kafka Connect and AMQP messaging.
• Make sure data reaches downstream consumers, applications and storage reliably.
• Maintain MongoDB collections, data structures and the application integrations that use them, in line with architecture guidance.
• Build, deploy and run cloud-native services, mainly on AWS, using Kubernetes and managed cloud services.
• Help modernise and move older data processing components onto Kafka-based streaming.
• Work with tech leads and engineering teams to turn architecture designs into solid, production-ready code.
• Investigate performance, reliability and data consistency problems in distributed data systems.
• Help improve code quality, automated testing, CI/CD practice and engineering processes.

Requirements

• Strong professional background in data engineering, especially streaming and distributed systems.
• Advanced Java programming (11+) with experience using the Spring framework.
• Hands-on work with Apache Kafka, including Kafka Streams, Kafka Connect and building Kafka integrations.
• Practical experience with a streaming or distributed processing framework such as Apache Flink, Apache Spark or Spark Streaming.
• A solid grasp of event-driven architecture, asynchronous communication patterns and messaging systems, including AMQP.
• Experience designing and building data transformation, enrichment and integration pipelines.
• Experience building and connecting microservices and distributed applications.
• Hands-on use of AWS services, particularly S3, SQS and EKS.
• Experience running containerised applications with Docker and Kubernetes.
• Working knowledge of MongoDB, including collections, document structures and Java data access.
• Familiarity with Redis and how it is used in distributed applications.
• Experience with Git workflows and CI/CD pipelines, ideally GitLab and GitLab CI/CD.
• Strong troubleshooting skills, especially in distributed, high-throughput data environments.
• Ability to work well with tech leads and cross-functional teams while owning implementation and delivery.
• Experience with Apache Pulsar or Apache Storm, especially in maintaining or modernising legacy systems.
• Familiarity with Microsoft Azure services and hybrid or multi-cloud setups.
• Experience processing telemetry, IoT or other event data from physical devices.
• Knowledge of SMTP-based integrations.
• Experience moving legacy messaging and stream processing systems to modern Kafka-based architectures.
• Understanding of performance tuning, fault tolerance, monitoring and observability in distributed streaming systems.

Conditions

Hybrid working with flexible working hours.

The company is committed to an open, inclusive workplace and hires on merit, skills, qualifications and potential. Ask for adjustments if you need support during recruitment.

Benefits

• Competitive salary package, share plan, company performance bonuses, value-based recognition awards and a referral bonus.
• Career coaching, global career opportunities, non-linear career paths and internal programmes for management and technical leadership.
• Complex projects, rotations, internal tech communities, training, certifications, coaching, online learning subscriptions, pass-it-on sessions, workshops and conferences.
• Employee assistance programme, a global wellbeing programme and access to wellbeing apps.
• Global internal tech communities, hobby clubs and interest groups, inclusion and diversity programmes, and events and celebrations.