Senior Data Engineer (Real-Time Data Processing & Streaming)

Description

The team is hiring a senior data engineer to design, build and maintain scalable, high-performance data processing systems.

The focus is real-time processing, streaming technologies and event-driven architectures, with pipelines that take in, transform, enrich and deliver data from connected physical devices to downstream applications and storage.

You will work with tech leads, software engineers and other teams in a cloud-native setting.

The company has spent over two decades in technology consulting, partnering with leading brands in many industries to build digital platforms.

Responsibilities

• Build and run real-time and near-real-time data pipelines that scale.
• Apply Apache Kafka, Apache Flink and Apache Spark to streaming workloads.
• Write and maintain Java services and integration components that process device data.
• Create complex transformations, mappings, validations and enrichments that meet business and technical needs.
• Build and support event-driven integrations with Kafka, Kafka Streams, Kafka Connect and AMQP messaging.
• Make sure data reaches downstream consumers, applications and storage reliably.
• Maintain MongoDB collections, data structures and application integrations in line with architecture guidelines.
• Deploy and operate cloud-native services, mainly on AWS, using Kubernetes and managed cloud services.
• Help modernise and migrate legacy data processing toward Kafka-based streaming.
• Turn architectural designs into robust, production-ready code with technical leads and engineering teams.
• Investigate performance, reliability and data consistency problems across distributed systems.
• Improve code quality, automated testing, CI/CD and engineering processes.

Requirements

• The following are required:
• Several years of professional data engineering experience, focused on streaming and distributed systems.
• Advanced Java (11+) skills, including Spring.
• Hands-on work with Apache Kafka, including Kafka Streams, Kafka Connect and Kafka-based integrations.
• Practical use of a streaming or distributed processing framework such as Apache Flink, Apache Spark or Spark Streaming.
• A solid grasp of event-driven design, asynchronous messaging patterns and messaging systems including AMQP.
• Experience designing and building transformation, enrichment and integration pipelines.
• Experience building and integrating microservices and distributed applications.
• Hands-on AWS experience, especially S3, SQS and EKS.
• Experience running containerised applications with Docker and Kubernetes.
• Working knowledge of MongoDB, including collections, document structures and Java data access.
• Familiarity with Redis and how it is used in distributed applications.
• Git-based workflows and CI/CD pipelines, ideally with GitLab and GitLab CI/CD.
• Strong troubleshooting skills, especially in distributed, high-throughput environments.
• Ability to work well with tech leads and cross-functional engineering teams while owning implementation and delivery.
• Experience with Apache Pulsar or Apache Storm, especially in legacy maintenance or modernisation work.
• Familiarity with Microsoft Azure and hybrid or multi-cloud setups.
• Experience processing telemetry, IoT or other event data from physical devices.
• Knowledge of SMTP-based integrations.
• Experience moving legacy messaging and stream processing to modern Kafka-based architectures.
• Understanding of performance tuning, fault tolerance, monitoring and observability in distributed streaming systems.

Conditions

Hybrid work with flexible working hours.

Hiring is based on merit, skills, qualifications and potential, and the company welcomes applicants from all backgrounds. Candidates who need adjustments during recruitment can ask for them.

Benefits

• Competitive salary package, share plan, company performance bonuses, value-based recognition awards and a referral bonus.
• Career coaching, global career opportunities, non-linear career paths and internal programmes for management and technical leadership.
• Complex projects, rotations, internal tech communities, training, certifications, coaching, online learning subscriptions, pass-it-on sessions, workshops and conferences.
• Employee assistance programme.
• Global wellbeing programme and access to wellbeing apps.
• Global internal tech communities, hobby clubs, interest groups, inclusion and diversity programmes, and events.