Senior Data Engineer (Real-Time Data Processing & Streaming)

Description

The company is hiring a senior data engineer to help design, build and maintain scalable, high-performance data processing systems.

The focus is real-time processing, streaming and event-driven design. You will work on pipelines that take in data from connected physical devices and pass it, after transforming and enriching it, to applications and storage.

You will work alongside Tech Leads, software engineers and other teams to deliver dependable, efficient and maintainable solutions in a cloud-native setup.

The company combines engineering skill, industry knowledge and a people-focused approach to build digital platforms for leading brands across many sectors. It has operated for over 20 years.

Responsibilities

• Design, build and maintain real-time and near-real-time data pipelines that scale.
• Build streaming solutions with Apache Kafka, Apache Flink and Apache Spark.
• Write and maintain Java services and integration components that process device data.
• Create complex transformations, mappings, validations and enrichments that meet business and technical needs.
• Build and run event-driven integrations with Kafka, Kafka Streams, Kafka Connect and AMQP messaging.
• Make sure data reaches downstream consumers, applications and storage systems reliably.
• Maintain MongoDB collections, data structures and related application integrations in line with architecture guidelines.
• Build, deploy and run cloud-native services, mainly on AWS, using Kubernetes and managed cloud services.
• Help modernise and migrate legacy data processing components to Kafka-based streaming.
• Work with technical leads and engineering teams to turn architecture designs into production-ready code.
• Investigate performance, reliability and data consistency problems in distributed data systems.
• Help improve code quality, automated testing, CI/CD practice and engineering processes.

Requirements

• Solid professional background in data engineering, especially streaming and distributed systems.
• Advanced Java skills (version 11 or later) and experience with the Spring framework.
• Hands-on work with Apache Kafka, including Kafka Streams, Kafka Connect and building Kafka integrations.
• Practical experience with a distributed processing framework such as Apache Flink, Apache Spark or Spark Streaming.
• A firm grasp of event-driven architecture, asynchronous communication patterns and messaging systems, including AMQP.
• Experience designing transformation, enrichment and integration pipelines.
• Experience building and connecting microservices and distributed applications.
• Hands-on use of AWS services, particularly S3, SQS and EKS.
• Experience deploying and running containerised applications with Docker and Kubernetes.
• Working knowledge of MongoDB, including collections, document structures and Java data access.
• Familiarity with Redis and how it is used in distributed applications.
• Experience with Git workflows and CI/CD pipelines, ideally GitLab and GitLab CI/CD.
• Strong troubleshooting skills, especially in distributed, high-throughput data environments.
• Ability to collaborate with technical leads and cross-functional engineering teams while owning implementation and delivery.
• Experience with Apache Pulsar or Apache Storm, especially in maintaining or modernising legacy systems.
• Familiarity with Microsoft Azure and hybrid or multi-cloud environments.
• Experience handling telemetry, IoT or other event data from physical devices.
• Knowledge of SMTP-based integrations.
• Experience moving legacy messaging and stream processing systems to modern Kafka-based architectures.
• Understanding of performance tuning, fault tolerance, monitoring and observability in distributed streaming systems.

Conditions

Hybrid working with flexible hours.

Hiring is based on merit, skills and potential, and adjustments are available on request during recruitment.

Benefits

• Competitive salary package, share plan, company performance bonuses, recognition awards for living company values, and a referral bonus.
• Career coaching, international career moves, non-linear career paths, and internal programmes for management and technical leadership.
• Complex projects, job rotations, internal tech communities, training, certifications, coaching, online learning subscriptions, peer-led knowledge sessions, workshops and conferences.
• Hybrid working, flexible hours and an employee assistance programme.
• Global wellbeing programme and access to wellbeing apps.
• Global internal tech communities, hobby clubs and interest groups, inclusion and diversity programmes, and company events and celebrations.