Senior Data Engineer

Description

The company is a German-American deep-tech business building the world's first commercial laser fusion power plant, using direct-drive laser fusion technology that builds on the 2021 fusion ignition result at Lawrence Livermore.

The role begins with a blank slate and carries work through to production, centred on cloud-native data platforms, near-real-time pipelines, and modernizing large Databricks and AWS infrastructure.

You will work on a team where data systems support fusion research and clean energy goals.

Responsibilities

• Build scalable, cloud-native data platforms, starting from nothing and taking them to production.
• Create near-real-time ingestion pipelines based on event-driven designs.
• Set and enforce platform standards covering lake and lakehouse principles, medallion architecture, and data contracts.
• Rework Spark and PySpark code so it runs faster and is easier to maintain.
• Bring code quality, testing, and CI/CD practices into data pipeline work.
• Lead uptake of AI tools and agentic workflows within the data engineering team.
• Keep data quality, observability, and reliability high across pipelines and platforms.
• Design and ship self-service tools and microservices that hide platform complexity.
• Work directly with the customer's scientific and engineering teams.
• Track platform performance, reliability, and cloud infrastructure spend.

Requirements

• 5+ years of data engineering experience.
• 3+ years running Databricks and Delta Lake in production.
• Strong grasp of medallion architecture and asynchronous, event-driven pipeline design.
• Advanced skills in Python, PySpark, and SQL.
• Hands-on depth with Databricks, including Unity Catalog, Lakeflow Jobs and Workflows, and job tuning.
• Strong AWS skills across S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway.
• Infrastructure as code and CI/CD using Terraform and Databricks Asset Bundles.
• Working knowledge of Git workflows, pull request reviews, and automated testing.
• Experience setting up secure CI pipelines with OIDC or service principals.
• Practical MLflow work: experiment tracking, model registration, and reproducibility.
• Solid understanding of data quality, lineage, provenance, schema contracts, and versioning.
• Used a data quality tool such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX.
• Know how to monitor and observe systems and control cloud costs.
• English at Upper-Intermediate level or above.
• A strong analytical mindset.
• Ownership and accountability in your work.
• Able to explain complex technical ideas clearly.
• A proactive approach to solving problems.
• Care about scalability and maintainability.
• A collaborative, team-oriented attitude.
• A plus: packaging workloads in Docker containers and running batch jobs on AWS.
• A plus: experiment optimization tools such as scikit-optimize, BoTorch, Ax, and scipy.
• A plus: building feature sets and train/test splits that prevent data leakage.
• A plus: work in regulated or high-security settings, including data residency requirements.
• A plus: experience disabling or restricting AI features in secured environments.