Senior Data Engineer

Description

The company is an experience innovation firm that works with large, well-known brands on digital transformation across data, AI, creative work and technology.

The role sits on projects where you build data platforms and pipelines that support analytics, AI and data products. You will work on a range of projects, from building new solutions to shaping customer experiences.

The company works across offices in several countries and uses a global, borderless way of working that supports collaboration between teams.

Responsibilities

• Design, build and optimise modern cloud data platforms that feed analytics, AI and data products.
• Look into the business problems behind the data, and use analysis and experience to suggest changes to requirements instead of only building what is specified.
• Build scalable batch, streaming and near-real-time pipelines that deliver high-quality curated datasets, with strong governance, security and observability.
• Support AI and GenAI systems by building pipelines for machine learning, causal modelling and LLM applications such as RAG and agent-based systems.
• Work closely with data scientists, ML engineers and platform teams.
• Work mainly on AWS and Databricks, with some projects on GCP or Azure/Fabric, including cloud migration programmes where you help re-platform pipelines and data models.
• Develop reliable batch, streaming and near-real-time pipelines with Spark and Delta Lake.
• Build ingestion, transformation and curation workflows for structured and unstructured data.
• Implement lakehouse patterns with bronze, silver and gold medallion layers, creating reusable, scalable systems that meet enterprise needs.
• Deliver model-ready, production-grade datasets for analytics, ML, causal modelling and optimisation.
• Build data flows for GenAI use cases, including LLMs, RAG and vector-based data, and for agent-based architectures.
• Use Databricks MLflow and Unity Catalog, or equivalent tools on other platforms.
• Design logical and physical data models for analytical and operational needs.
• Orchestrate workflows with Databricks Workflows, Airflow, dbt or equivalents such as Azure Data Factory or Cloud Composer, focusing on automation, reliability and maintainability.
• Apply event-driven and streaming architecture patterns.
• Ensure governance, lineage, quality and access control (RBAC/ABAC) using Unity Catalog, AWS Lake Formation or equivalents such as Microsoft Purview or GCP Dataplex.
• Monitor data freshness, pipeline reliability and SLA adherence.
• Build serving layers, such as APIs, feature inputs and analytical endpoints, for downstream ML and AI platforms.
• Continuously improve pipeline and infrastructure performance, scalability and cost.

Requirements

• Ask the right questions to understand why stakeholders want something, not just what they want.
• Identify gaps and better alternatives rather than delivering a brief as written.
• Work closely with data scientists, ML engineers, analysts and business stakeholders.
• Help drive adoption of data products and contribute to best practice.
• Strong hands-on experience with Apache Spark and Delta Lake.
• Strong Python and SQL skills.
• Proven experience building batch and streaming pipelines and production-grade data platforms.
• Solid understanding of data modelling, data quality and governance.
• Key requirement: strong, demonstrable hands-on experience with the Databricks Lakehouse Platform, ideally on AWS (for example S3, Glue, EMR, Redshift, Kinesis or Lambda).
• Deep expertise in at least one major cloud (AWS, GCP or Azure/Fabric) and demonstrable ability and openness to work across clouds.
• Candidates with a strong Azure/Fabric background (Data Factory, Synapse, Fabric Lakehouse, Azure Databricks) are encouraged to apply if they are keen to work in other clouds.
• GCP experience (for example BigQuery, Dataflow, Pub/Sub, Cloud Storage or Dataproc) is highly valued, especially for migrations.
• Experience with lakehouse architectures and distributed systems.
• Strong understanding of scalability, reliability and performance.
• Naturally curious, with strong problem-solving skills and a collaborative approach in cross-functional teams.
• Willing to challenge business requirements rather than accept them at face value.
• Adaptable and eager to work across platforms. Agile or consulting experience is a plus.
• Hands-on experience across more than one cloud, or cloud migration experience (for example Azure to GCP). Nice to have.
• Snowflake experience. Nice to have.
• GenAI and AI data systems experience, such as RAG pipelines, vector databases and LLM data preparation. Nice to have.
• CI/CD for data pipelines and infrastructure-as-code (Terraform, CloudFormation). Nice to have.
• Streaming technologies (for example Kafka), Spark optimisation or ML and experimentation platforms. Nice to have.
• Experience building data products or large-scale analytics platforms. Nice to have.

Conditions

The company does not require age, gender, marital status or a photo, and reviews all candidates on skills, experience and potential.

The company is committed to inclusion and accessibility, and will make reasonable adjustments in the interview process on request.

Benefits

• Private health insurance for all employees.
• Education programme with training and certification.
• Wellbeing programme.
• Free coffee, drinks and snacks at the office, plus company dinners.
• Regular social events such as ski trips, go-karting, laser tag, wine tasting, picnics and cooking classes.
• Competitive salary and 24 days of vacation.
• Annual company events with the whole team.
• Challenging projects.
• An open feedback culture.