Data Engineer Lead

Description

This is a hands-on engineering lead role in the Data Engineering organization of a global commercial and personal insurer, sitting in the Data function. You will design, build and run production data pipelines and platforms, and apply sound engineering judgment to each solution.

The team uses GenAI tools such as Claude Code and Snowflake Cortex to speed up work across the development cycle. The team wants someone who enjoys learning, tracks a fast-changing field, and looks for new ways to use these tools.

You will work with the product owner, business stakeholders and other engineers. Clear communication and analytical thinking count as much as technical depth, including asking good questions, weighing trade-offs and explaining your reasoning.

The team values documentation and design reviews, mentors people at every level, and keeps business stakeholders involved in the work.

Responsibilities

• Build and run production PySpark and Python pipelines on AWS EMR, Glue and S3, with built-in data quality checks and observability.
• Own data products from requirements through production, using Agile/Scrum with sprint planning, CI/CD, release management and production readiness checks.
• Turn business and product needs into technical designs that define pipeline structure, data models and SLAs.
• Apply architecture and design patterns, weigh design trade-offs and explain your reasoning to peers and stakeholders.
• Set and enforce team standards for coding, data quality, reusable pipeline patterns and observability.
• Carry out thorough code reviews and pair with engineers on difficult problems to raise the quality of the whole team.
• Use GenAI tools such as Claude Code CLI and Snowflake Cortex across the development cycle to write better code faster.
• Work with product owners to refine requirements, expose trade-offs and push back when scope cannot be built well.
• Explain design choices and delivery progress clearly to both engineers and non-technical stakeholders.
• Mentor junior engineers through reviews, pairing and knowledge sharing while delivering product work.
• Write technical documentation as a routine part of delivery: Architecture Decision Records (ADRs), runbooks, data dictionaries and pipeline lineage notes.
• Find and roll out productivity tools and good practices, and set team norms for using them safely and effectively.

Requirements

• Bachelor's degree in computer science, information systems, engineering or a closely related technical field.
• 7+ years of hands-on data engineering, with a steady record of delivering production pipelines from start to finish.
• Strong skills in PySpark, Python and SQL, including Spark tuning, partition management and Snowflake query performance tuning.
• Practical experience with AWS data services: S3, Glue, EMR, Aurora, Lambda, IAM and CloudWatch.
• Working knowledge of Lakehouse patterns, data modeling and fault-tolerant ingestion, with proof you have applied them.
• Ability to break complex requirements into parts, reason through design trade-offs and reach practical, well-argued solutions.
• Able to explain technical decisions in plain language to both engineers and business stakeholders.
• Motivated to keep learning and to stay current with changing tools, frameworks and engineering practices.
• Demonstrated use of GenAI tools, such as Claude Code CLI or Snowflake Cortex, to improve productivity or solution quality.
• Nice to have: AWS certification such as Solutions Architect Associate or an equivalent.
• Nice to have: familiarity with insurance or financial services data, such as claims, policy, risk or actuarial structures.
• Nice to have: experience writing ADRs, design notes or technical documentation within a collaborative engineering team.

Conditions

Primarily based in the office, with hybrid flexibility.

Base salary range is $125,000-135,000, and the role is eligible for a bonus under the applicable incentive plan.

Actual pay depends on skills, experience and qualifications, and candidates whose expectations are above the posted range are still encouraged to apply.

Equal opportunity is provided to all qualified people regardless of race, color, religion, age, gender, gender expression, national origin, veteran status, disability or other legally protected categories.

Benefits

• Total Rewards program with health, wellbeing, financial security and professional development benefits.
• Continuous learning and development opportunities.
• Employee Resource Groups across global chapters that support inclusion and belonging.