Data Engineer – AWS Kinesis & Databricks - AU Citizen

Description

The team is hiring a hands-on data engineer to build dependable real-time data pipelines on Amazon Kinesis and Databricks.

You will help move high-volume data from Kinesis into the Databricks Lakehouse, using Spark Structured Streaming and Delta tables.

Responsibilities

• Build and maintain real-time pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Write streaming logic in Databricks with Spark Structured Streaming.
• Ingest and manage streaming data in Delta tables.
• Tune pipelines for speed, latency, reliability and cost.
• Set up monitoring and alerts with AWS CloudWatch.
• Coordinate with batch data engineers so streaming and batch pipelines work together.
• Write clean, testable Python and SQL.
• Diagnose pipeline and data-processing problems and help improve how the team works.

Requirements

• Production experience with Amazon Kinesis Data Streams and/or Kinesis Data Firehose is essential.
• Strong Python and SQL skills.
• Experience with Apache Spark Structured Streaming.
• Hands-on work with Databricks, ideally including Delta Lake and Unity Catalog.
• Good working knowledge of AWS services, especially IAM, S3 and CloudWatch.
• Understanding of how to design scalable, reliable data pipelines.
• Strong troubleshooting and problem-solving skills.
• Kafka experience is a plus.

Conditions

Only Australian citizens who hold an active security clearance can apply, because the role handles sensitive information.

The company values equal opportunity and wants every team member cleared to handle sensitive work.

Benefits

• 18 weeks of paid parental leave.
• Long and short career breaks.
• A structured career development program.
• Local and international career moves.
• Certified as a Family Inclusive Workplace.
• Flexible work arrangements.
• Placed in the top 3 of last year's Diversity & Inclusion Index.