Senior Data Engineer (Real-Time Data Processing & Streaming)

Description

The company builds real-time data processing and streaming systems, and this role focuses on designing, building and running scalable, high-performance pipelines.

Your work centres on pipelines that bring in data from connected physical devices, transform and enrich it, and pass it to downstream apps and storage.

You will work with tech leads, software engineers and other teams to turn architecture into dependable, efficient, maintainable data systems in the cloud.

The company consults for major brands across many industries, building digital platforms and intelligent digital experiences.

Responsibilities

• Build and upkeep real-time and near-real-time data pipelines that scale.
• Use Apache Kafka, Apache Flink and Apache Spark to implement streaming data processing.
• Write and maintain Java services and integration components that handle data from physical devices.
• Create complex transformations, mappings, validations and enrichments that meet business and technical needs.
• Build event-driven integrations with Kafka, Kafka Streams, Kafka Connect and AMQP messaging.
• Make sure data reaches consumers, applications and storage systems reliably.
• Maintain MongoDB collections, data structures and the application integrations around them, in line with architecture guidelines.
• Deploy and run cloud-native services, mainly on AWS, using Kubernetes and managed cloud services.
• Help modernise and migrate legacy data processing toward Kafka-based streaming.
• Turn architectural designs into robust, production-ready code with tech leads and engineering teams.
• Investigate performance, reliability and data consistency problems across distributed systems.
• Improve code quality, automated testing, CI/CD and engineering processes.

Requirements

• Strong professional background in data engineering, especially streaming and distributed systems.
• Advanced Java 11+ skills, including the Spring framework.
• Hands-on Apache Kafka work with Kafka Streams, Kafka Connect and Kafka-based integrations.
• Practical use of a streaming or distributed framework: Apache Flink, Apache Spark or Spark Streaming.
• Solid grasp of event-driven design, asynchronous communication and messaging, including AMQP.
• Experience building data transformation, enrichment and integration pipelines.
• Experience building and integrating microservices and distributed applications.
• Hands-on AWS experience, particularly S3, SQS and EKS.
• Experience running containerised applications with Docker and Kubernetes.
• Working knowledge of MongoDB, including collections, document structures and Java data access.
• Familiarity with Redis and how it is used in distributed applications.
• Git-based workflows and CI/CD pipelines, preferably GitLab and GitLab CI/CD.
• Strong troubleshooting skills in distributed, high-throughput data environments.
• Collaborate well with tech leads and cross-functional teams while owning implementation and delivery.
• Experience with Apache Pulsar or Apache Storm, especially in legacy maintenance or modernisation.
• Familiarity with Microsoft Azure and hybrid or multi-cloud setups.
• Experience with telemetry, IoT or other event data from physical devices.
• Knowledge of SMTP-based integrations.
• Experience moving legacy messaging and stream processing to modern Kafka architectures.
• Understanding of performance tuning, fault tolerance, monitoring and observability in distributed streaming systems.

Conditions

Hybrid working with flexible hours.

Equal opportunity: hiring is based on merit, skills and potential, and adjustments can be requested during recruitment.

Benefits

• Pay: competitive salary, share plan, company performance bonuses, value-based recognition awards and a referral bonus.
• Career: coaching, global career opportunities, non-linear career paths and internal programmes for management and technical leadership.
• Learning: complex projects, rotations, internal tech communities, training, certifications, coaching, online learning subscriptions, pass-it-on sessions, workshops and conferences.
• Wellbeing: a global wellbeing programme, an employee assistance programme and wellbeing apps.
• Community: global tech communities, hobby clubs, interest groups, inclusion and diversity programmes, and events.