Senior Data Engineer

Description

The team is building a clean energy deep-tech initiative around a commercial laser fusion power plant, and this role focuses on cloud-native data platforms that are new builds and need to run in production.

The company is a German-American deep-tech firm developing direct-drive laser fusion technology, building on the 2021 fusion ignition result at Lawrence Livermore.

You will work on greenfield data projects, build near-real-time pipelines, and modernize large data infrastructure on Databricks and AWS, with work done at the company for the company's team.

Responsibilities

• Build scalable cloud-native data platforms, from greenfield start to production.
• Design and implement near-real-time ingestion pipelines based on event-driven patterns.
• Set and enforce data platform standards covering Data Lake and Lakehouse principles, medallion architecture, and data contracts.
• Refactor and optimize Spark and PySpark scripts for speed and easier maintenance.
• Bring in code quality, testing, and CI/CD practices across data pipelines.
• Lead adoption of AI tooling and agentic workflows within the data engineering team.
• Keep data quality, observability, and reliability in check across pipelines and platforms.
• Build self-service tools and microservices that hide platform complexity.
• Work directly with the customer's scientific and engineering teams.
• Track platform performance, reliability, and cloud infrastructure costs.

Requirements

• 5+ years of data engineering experience.
• 3+ years of production work with Databricks and Delta Lake.
• Strong grasp of medallion architecture and asynchronous, event-driven pipeline design.
• Advanced skills in Python, PySpark, and SQL.
• Deep hands-on Databricks experience, including Unity Catalog, Lakeflow Jobs/Workflows, and job tuning.
• Strong AWS skills across S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway.
• Hands-on infrastructure as code and CI/CD with Terraform and Databricks Asset Bundles.
• Practical experience with Git workflows, pull request reviews, and automated testing.
• Experience setting up secure CI pipelines using OIDC or service principals.
• Practical MLflow experience covering experiment tracking, model registration, and reproducibility.
• Solid understanding of data quality, lineage, provenance, schema contracts, and versioning.
• Experience with a data quality framework such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX.
• Experience with monitoring, observability, and cloud cost optimization.
• Upper-Intermediate English or higher.
• A plus: experience packaging workloads in Docker containers and running batch jobs on AWS.
• A plus: familiarity with experiment optimization tools such as scikit-optimize, BoTorch, Ax, and scipy.
• A plus: experience building feature sets and train/test splits that prevent data leakage.
• A plus: experience in regulated or high-security environments, including data residency requirements.
• A plus: experience disabling or restricting AI features in secured environments.
• Strong analytical thinking.
• Ownership and accountability.
• Clear communication of complex technical concepts.
• A proactive approach to solving problems.
• Focus on scalability and maintainability.
• A collaborative, team-first attitude.

Conditions

Equal opportunity policy applies.