Senior Data Engineer (Real-Time Data Processing & Streaming)

Description

The company is hiring an experienced senior data engineer to help design, build and maintain scalable, high-performance data processing systems.

The focus is real-time processing, streaming technologies and event-driven architectures, on pipelines that take in, transform, enrich and deliver data from connected physical devices to downstream applications and storage.

You will work with tech leads, software engineers and other cross-functional teams in a cloud-native setup.

The company consults for leading brands across industries, building digital platforms and intelligent experiences, and works on prototypes that go on to real-world use.

Responsibilities

• Build and maintain scalable pipelines for real-time and near-real-time data processing.
• Create streaming data solutions with Apache Kafka, Apache Flink and Apache Spark.
• Write and maintain Java services and integration components that process data from physical devices.
• Implement complex transformations, mappings, validations and enrichments that meet business and technical needs.
• Build and maintain event-driven integrations with Kafka, Kafka Streams, Kafka Connect and messaging protocols such as AMQP.
• Make sure data reaches downstream consumers, applications and storage systems reliably.
• Maintain MongoDB collections, data structures and related application integrations, following architecture guidelines.
• Build, deploy and run cloud-native services, mainly on AWS, using Kubernetes and managed cloud services.
• Help modernise and migrate legacy data processing components to Kafka-based streaming.
• Turn architecture designs into robust, production-ready code with technical leads and engineering teams.
• Investigate performance, reliability and data consistency problems across distributed data systems.
• Contribute to code quality, automated testing, CI/CD and ongoing improvement of engineering processes.

Requirements

• Required skills and experience:
• Solid professional background in data engineering, focused on streaming data and distributed systems.
• Advanced Java (11+) skills, including the Spring framework.
• Hands-on work with Apache Kafka, including Kafka Streams, Kafka Connect and building Kafka-based integrations.
• Practical experience with Apache Flink, Apache Spark or Spark Streaming.
• Good understanding of event-driven design, asynchronous communication and messaging systems, including AMQP.
• Experience designing and building data transformation, enrichment and integration pipelines.
• Experience building and integrating microservices and distributed applications.
• Hands-on experience with AWS services, especially S3, SQS and EKS.
• Experience running containerised applications with Docker and Kubernetes.
• Working knowledge of MongoDB, including collections, document structures and Java data access.
• Familiarity with Redis and how it is used in distributed applications.
• Experience with Git-based workflows and CI/CD pipelines, ideally GitLab and GitLab CI/CD.
• Strong troubleshooting and problem-solving skills, especially in distributed, high-throughput environments.
• Ability to work well with tech leads and cross-functional teams while owning implementation and delivery.
• Experience with Apache Pulsar or Apache Storm, particularly in maintaining or modernising legacy systems.
• Familiarity with Microsoft Azure and hybrid or multi-cloud environments.
• Experience processing telemetry, IoT or other event data from physical devices.
• Knowledge of SMTP-based integrations.
• Experience migrating legacy messaging and stream processing systems to modern Kafka-based architectures.
• Understanding of performance tuning, fault tolerance, monitoring and observability in distributed streaming systems.

Conditions

Hybrid work with flexible working hours.

The company is committed to an open, inclusive workplace and hires on merit, skills and potential.

Adjustments or support during recruitment are available on request.

Benefits

• Pay: competitive salary package, share plan, company performance bonuses, value-based recognition awards and a referral bonus.
• Career: career coaching, global career opportunities, non-linear career paths and internal programmes for management and technical leadership.
• Learning: complex projects, rotations, internal tech communities, training, certifications, coaching, online learning platform subscriptions, pass-it-on sessions, workshops and conferences.
• Wellbeing: a global internal wellbeing programme, access to wellbeing apps and an employee assistance programme.
• Community: global internal tech communities, hobby clubs and interest groups, inclusion and diversity programmes, and events and celebrations.