Senior Data Engineer (Real-Time Data Processing & Streaming)

Description

The company is hiring a senior data engineer to help design, build and run data processing systems that scale and perform well.

The focus is real-time processing, streaming and event-driven design. You will build pipelines that take in, reshape, enrich and pass on data from connected physical devices to applications and storage.

You will work day to day with tech leads, software engineers and other teams to deliver dependable, efficient, maintainable solutions in a cloud-native setting.

The company consults for major brands across many industries, building digital platforms and reshaping their businesses through engineering. It has been operating for over 20 years.

Responsibilities

• Build and support real-time and near-real-time data pipelines that scale.
• Create streaming solutions with Apache Kafka, Apache Flink and Apache Spark.
• Write and maintain Java services and integration components that handle data from physical devices.
• Carry out complex transformations, mappings, validation and enrichment to meet business and technical needs.
• Set up event-driven integrations with Kafka, Kafka Streams, Kafka Connect and AMQP messaging.
• Make sure data reaches downstream consumers, applications and storage reliably.
• Build and run MongoDB collections, data structures and the application integrations that use them, in line with architecture guidance.
• Build, deploy and run cloud-native services mainly on AWS, using Kubernetes and managed cloud services.
• Help modernize and migrate older data processing parts onto Kafka-based streaming.
• Work with tech leads and engineering teams to turn architecture designs into robust production code.
• Investigate performance, reliability and data consistency problems across distributed systems.
• Improve code quality, automated testing, CI/CD practices and engineering processes.

Requirements

• A solid professional background in data engineering, focused on streaming and distributed systems.
• Advanced Java 11+ skills, including the Spring framework.
• Hands-on Kafka work, including Kafka Streams, Kafka Connect and Kafka-based integrations.
• Practical use of a streaming or distributed processing framework: Apache Flink, Apache Spark or Spark Streaming.
• A good grasp of event-driven design, asynchronous communication and messaging, including AMQP.
• Experience designing and building transformation, enrichment and integration pipelines.
• Experience building and integrating microservices and distributed applications.
• Hands-on AWS experience, especially S3, SQS and EKS.
• Experience running containerized applications with Docker and Kubernetes.
• Working knowledge of MongoDB, including collections, document structures and Java-based data access.
• Familiarity with Redis and how it is used in distributed applications.
• Experience with Git workflows and CI/CD pipelines, preferably GitLab and GitLab CI/CD.
• Strong troubleshooting and problem solving, especially in distributed, high-throughput environments.
• Able to work well with tech leads and cross-functional engineering teams while owning implementation and delivery.
• Experience with Apache Pulsar or Apache Storm, including legacy system maintenance or modernization.
• Familiarity with Microsoft Azure and hybrid or multi-cloud setups.
• Experience handling telemetry, IoT or other event data from physical devices.
• Knowledge of SMTP-based integrations.
• Experience moving legacy messaging and stream processing to modern Kafka architectures.
• Understanding of performance tuning, fault tolerance, monitoring and observability in distributed streaming systems.

Conditions

Hybrid work with flexible working hours.

Hiring is based on merit, skills, qualifications and potential, and the company welcomes applicants from all backgrounds; adjustments during recruitment are available on request.

Benefits

• Competitive salary package and share plan.
• Company performance bonuses, value-based recognition awards and a referral bonus.
• Career coaching, global career opportunities, non-linear career paths and internal development programmes for management and technical leadership.
• Complex projects, rotations, internal tech communities, training, certifications, coaching, online learning platform subscriptions, peer-led sessions, workshops and conferences.
• Employee assistance programme, a global internal wellbeing programme and access to wellbeing apps.
• Global internal tech communities, hobby clubs, interest groups, inclusion and diversity programmes, and events and celebrations.