Senior Data Engineer with Python and Databricks

Description

The company is a consultancy that has spent more than twenty years using technology to drive change, combining strong engineering, sector knowledge and a people-first approach.

It works with well-known brands across many industries to build digital platforms and intelligent customer experiences.

The role is a senior data engineering position on a team that takes work from prototype to real-world impact.

Responsibilities

• Work with data scientists and analysts to pin down their data needs and build effective pipelines to meet them.
• Design, build and maintain pipelines for ingesting, processing and transforming data using cloud services and other tools.
• Create and run data solutions in Azure using any of Data Factory, Synapse, Databricks/Spark, Polars/Pandas and Fabric.
• Put data validation and cleansing steps in place to protect the quality, integrity and reliability of the data.
• Make pipelines more scalable, efficient and cost-effective.
• Monitor pipelines, fix problems as they appear, and keep the data consistent and available.

Requirements

• Strong working knowledge of Python and SQL.
• Understanding of sound data design, including Kimball methods and star schemas, along with data warehousing and data modelling.
• Solid experience with relational databases and, ideally, NoSQL databases.
• Strong grasp of ETL and ELT processes.
• Exposure to data monitoring and observability tools.
• Relevant experience with event-driven architecture.
• Good understanding of Agile ways of working.

Conditions

Hybrid working with flexible hours.

An equal opportunities statement: the company welcomes applicants from all backgrounds and makes hiring decisions on merit, and it will support candidates who need adjustments during recruitment.

Benefits

• Competitive pay package, a share plan, company performance bonuses, value-based recognition awards and a referral bonus.
• Career coaching, global career moves, non-linear career paths, and internal programmes for management and technical leadership.
• Learning through complex projects, rotations, internal tech communities, training, certifications, coaching, online learning subscriptions, pass-it-on sessions, workshops and conferences.
• Employee assistance programme, a global internal wellbeing programme and access to wellbeing apps.
• Global internal tech communities, hobby clubs and interest groups, inclusion and diversity programmes, and events and celebrations.