Data Engineer

Description

Join a project in the fusion energy field that builds cloud-native data platforms for a deep-tech client on a large initiative.

The team runs a cloud data platform that supports scientific research, analytics and operational workflows, with a focus on data quality, lineage, governance, reproducibility and reliability.

You will support experiments, analytics and machine learning work with scientists and engineers in an R&D setting.

Responsibilities

• Build and maintain scalable, reliable data platforms and pipelines for analytics and operational needs.
• Create distributed processing workflows with Apache Spark, pySpark and Databricks.
• Develop cloud-native data solutions on Azure and AWS.
• Design and tune data models for analytics, reporting and downstream use.
• Bring together data from many heterogeneous sources while keeping quality, consistency and traceability intact.
• Put quality checks, schema validation, lineage tracking and monitoring in place.
• Improve the performance, scalability, reliability and cost efficiency of pipelines and infrastructure.
• Work with scientists, analysts and engineers to turn business and research needs into technical solutions.
• Help with CI/CD, infrastructure automation and engineering practices.
• Explore and adopt new tools and methods in the data engineering field.

Requirements

• 3+ years of commercial data engineering experience.
• 2+ years of production work with Databricks.
• Solid Delta Lake skills, including medallion architecture and table optimization.
• Advanced Python, pySpark and SQL.
• Working knowledge of Databricks Unity Catalog, Lakeflow Jobs / Workflows and job optimization.
• Hands-on AWS experience with S3, IAM and KMS, plus at least one of AWS Batch, ECR, EventBridge, Lambda or API Gateway.
• Experience designing event-driven and asynchronous data pipelines.
• Strong grasp of Infrastructure as Code and CI/CD practices.
• Practical experience with Terraform and Databricks Asset Bundles.
• Experience with Git workflows, pull request reviews and automated testing.
• Understanding of secure authentication approaches such as OIDC and service principals.
• Practical MLflow experience, covering experiment tracking, model registry and reproducibility.
• Solid understanding of data quality, lineage, schema contracts and provenance.
• Experience with a data quality framework such as Great Expectations, Soda, pyDeequ or Databricks DQX.
• Experience monitoring platform reliability and cloud costs, and optimizing both.
• Upper-Intermediate English or higher.
• A plus: experience packaging workloads with Docker and running batch workloads on AWS.
• A plus: familiarity with ECR and containerized data processing.
• A plus: experience with experiment optimization libraries such as scikit-optimize, BoTorch, Ax or scipy.
• A plus: understanding of ML feature engineering and leakage prevention.
• A plus: experience in regulated or high-security environments.
• A plus: knowledge of data governance, compliance and data residency requirements.
• A plus: experience disabling or restricting AI-assisted tools and features in secure environments.