Lead Data Engineer

Description

The company helps clients design, build and run reliable data platforms that convert intricate information into business insight. It has been operating for over 20 years and is growing its Australian Data & AI capability.

The work covers the full data lifecycle: architecture, source integration, pipelines, semantic models, visualisation and reporting. Python and SQL are core skills, along with hands-on use of at least one of Databricks, Snowflake or Microsoft Fabric.

You will sit between clients, analysts, data scientists and engineers, turning business needs into solutions that are secure, maintainable and scalable. You will also guide other engineers and shape the team's engineering practices.

Responsibilities

• Meet clients directly to convert business requirements into technical designs, offering advice and input on design choices.
• Build and upkeep pipelines that take in, transform and deliver data from varied sources, including APIs.
• Build solutions on Databricks, Snowflake or Microsoft Fabric, picking the approach that fits each client's needs and current setup.
• Help shape data architecture, covering storage, integration and modelling.
• Create data models, ontologies and semantic layers with consistent business definitions for analytics and reporting.
• Set up orchestration, scheduling, monitoring and day-to-day operational support so data services stay dependable.
• Diagnose data and pipeline faults, and improve speed, scale and cost.
• Put data quality checks, automated tests and validation in place at every stage.
• Implement role-based access control and apply sound security practice across platforms and integrations.
• Apply DevOps methods to data work: version control, repository management, CI/CD and automated deployment.
• Produce data visualisations and reports that business users can readily understand and act on.
• Write documentation, review code and share knowledge to keep delivery maintainable and improving.

Requirements

• Hands-on experience delivering production data solutions with one or more of Databricks, Snowflake or Microsoft Fabric.
• Strong Python and SQL programming skills.
• Track record designing, building and running data pipelines, covering ETL/ELT, orchestration and monitoring.
• Solid grasp of data architecture and modelling, including analytical and dimensional approaches.
• Understanding of ontologies and semantic modelling, and how they link data structures to business concepts.
• Practical skill in tuning data processing and troubleshooting performance across pipelines, queries and platforms.
• Experience working with APIs, including authentication, error handling and dependable data exchange.
• Experience with RBAC and data security practice.
• Experience with Git, repository upkeep, automated testing and CI/CD for data solutions.
• Experience building data visualisations and reports.
• Ability to explain technical choices clearly, collaborate with business and technical stakeholders, and guide other engineers.
• Useful additional experience with PostgreSQL, SQL Server, Google BigQuery or Amazon Redshift.
• Useful additional experience with R or other programming languages.
• Useful additional experience with relevant cloud services on Microsoft Azure, AWS or Google Cloud.
• Useful additional experience: relevant certifications in Databricks, Snowflake or Microsoft Fabric.

Conditions

Hybrid working, plus 20 days per year of working from anywhere on top of the hybrid policy.

Loyalty leave: an extra day off on your 3rd, 4th and 5th work anniversaries, up to 3 extra days per year.

Parental leave: 12 weeks of primary carer's leave and 4 weeks of secondary carer's leave.

The company is an equal-opportunity employer that makes hiring decisions on merit.

Benefits

• Monthly lifestyle allowance towards health and wellbeing activities, such as gym memberships.
• Novated leasing: pre-tax car leasing for new or used cars.
• Employee assistance programmes.
• Non-linear career paths, with the option to move across domains, disciplines or geographies.
• Ongoing learning through certifications, internal academies, tech communities and funded conferences.