Lead Data Engineer

Description

The team helps clients plan, build and run dependable data platforms, so complex data becomes usable business insight.

The role covers the full data lifecycle: architecture, source integration, pipelines, semantic models, visualisation and reporting. You will need solid Python and SQL skills and hands-on work with at least one of Databricks, Snowflake or Microsoft Fabric.

You will work with clients, analysts, data scientists and other engineers to turn business needs into secure, maintainable, scalable data solutions. You will give technical direction, shape design choices and help raise the team's engineering standards.

The company has operated for 20+ years, combining engineering with industry expertise for global brands. It is expanding its Australian Data & AI capability.

Responsibilities

• Build, run and improve pipelines that ingest, transform and deliver data from varied sources, including APIs.
• Choose between Databricks, Snowflake and Microsoft Fabric based on each client's needs and current setup.
• Help shape data architecture, covering storage, integration and modelling.
• Create data models, ontologies and semantic layers with consistent business definitions for analytics and reporting.
• Set up orchestration, scheduling and monitoring, and provide operational support so data services stay reliable.
• Diagnose data and pipeline faults, and improve performance, scalability and cost efficiency.
• Put data quality checks, automated tests and validation in place at every stage.
• Implement role-based access control and apply security best practices to platforms and integrations.
• Apply DevOps to data work: version control, repository management, CI/CD and automated deployment.
• Produce visualisations and reports that business users find clear and useful.
• Write documentation, review code and share knowledge to keep delivery maintainable and improving.

Requirements

• Proven experience running production data solutions on one or more of Databricks, Snowflake or Microsoft Fabric.
• Strong Python and SQL programming skills.
• A track record designing, building and operating pipelines, covering ETL/ELT, orchestration and monitoring.
• A solid grasp of data architecture and modelling, including analytical and dimensional modelling.
• An understanding of ontologies and semantic modelling, and how they link data structures to business concepts.
• Hands-on skill in tuning data processing and fixing pipeline, query and platform performance problems.
• Experience integrating data through APIs, including authentication, error handling and dependable data exchange.
• Experience with role-based access control (RBAC) and data security practice.
• Experience with Git, repositories, automated testing and CI/CD for data solutions.
• Experience building data visualisations and reports.
• The ability to explain technical decisions clearly, work with business and technical stakeholders, and mentor other engineers.
• Useful additional experience: PostgreSQL, SQL Server, Google BigQuery or Amazon Redshift.
• Useful additional experience: R or other programming languages.
• Useful additional experience: relevant cloud services across Microsoft Azure, AWS or Google Cloud.
• Useful additional experience: relevant certifications in Databricks, Snowflake or Microsoft Fabric.

Conditions

Hybrid working, plus 20 days per year where you can work from anywhere on top of the hybrid policy.

The company is an equal-opportunity employer that hires on merit and will make adjustments during recruitment if you ask.

Benefits

• A monthly lifestyle allowance for health and wellbeing activities, such as gym memberships.
• Pre-tax novated leasing for new or used cars.
• Loyalty leave: an extra day on your 3rd, 4th and 5th work anniversaries, up to 3 extra days of leave per year.
• Parental leave: 12 weeks of primary carer's leave and 4 weeks of secondary carer's leave.
• Employee assistance programmes.
• Non-linear career progression, including moves between domains, disciplines or geographies.
• Learning support through certifications, internal academies, tech communities and funded conferences.