Senior Data Engineer (Databricks Migration)

Description

The team is moving a large analytics platform from BigQuery to Databricks, and you will take part in that move.

You will join the company's Data Engineering Center of Excellence, working on analytics for the retail sector.

The client is a retail technology company that builds AI-driven pricing tools for enterprise retailers, using analytics and algorithms to support pricing decisions.

The platform handles high-volume retail data, machine learning workloads, analytics pipelines and client-specific business logic. It is being rebuilt on a Lakehouse design using Spark, Delta Lake, medallion layers and Unity Catalog governance.

You will work with international teams and help shape architecture and data solutions.

Responsibilities

• Help move the platform from BigQuery to Databricks.
• Review existing ETL and ELT jobs and decide how each one should be migrated.
• Design and build scalable Lakehouse architectures with Databricks and Delta Lake.
• Build and maintain PySpark-based processing, and develop and tune pipelines for large retail and analytics datasets.
• Set up incremental processing and reusable transformation frameworks.
• Maintain the Bronze, Silver and Gold medallion layers.
• Apply data governance and security practices through Unity Catalog.
• Work with Data Science, Analytics, Product and Customer Engineering teams.
• Take part in architecture discussions and design technical solutions.
• Create reusable data platform components and engineering standards.
• Review code and help keep the platform reliable and easy to maintain.
• Diagnose and tune complex SQL and Spark workloads.
• Support production releases and modernization work.

Requirements

• 5+ years of professional experience as a Data Engineer.
• Strong Python skills and advanced SQL.
• Hands-on commercial experience with Databricks.
• Strong command of Apache Spark, mainly PySpark.
• Experience designing and building modern cloud-based data platforms.
• Experience building ETL or ELT pipelines and large-scale data processing solutions.
• Hands-on experience with Delta Lake.
• Experience with Spark Declarative Pipelines.
• Experience monitoring clusters, analyzing metrics and tuning performance.
• A good grasp of distributed data processing architectures.
• Solid knowledge of data warehousing and dimensional modeling.
• Experience with Airflow or a similar orchestration tool.
• Experience optimizing complex analytical SQL workloads.
• Experience setting up CI/CD for data engineering platforms.
• Strong troubleshooting and performance optimization skills.
• Ability to work well in cross-functional international teams.
• English at Upper-Intermediate level or higher.
• An analytical mindset and strong problem-solving skills.
• A proactive, ownership-driven approach.
• Able to work independently as well as within a team.
• Good communication and stakeholder collaboration skills.
• Enthusiasm for scalable data engineering and modern data platforms.
• Curiosity about continuous learning and new technology.
• Nice to have (a plus): experience with GCP cloud services.
• Nice to have (a plus): experience with AWS or Azure cloud platforms.
• Nice to have (a plus): experience in retail analytics or pricing optimization.
• Nice to have (a plus): experience supporting machine learning or AI-related data workloads.
• Nice to have (a plus): experience with platform modernization and cloud migration initiatives.

Conditions

Collaboration takes place within international teams.

Benefits

• Opportunities for continuous learning, technology growth and meaningful engineering impact on complex international projects.