Lead Cloud Engineer

Description

Our Purpose

Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.

Title and Summary

Interested in building the next generation cloud data ecosystem that processes petabytes of data and powers AI-driven insights for some of the world's largest brands?

The Marketing Services Technology team is transforming its analytics and data platform from traditional on-premises Hadoop environments to a modern cloud-native architecture powered by AWS, Databricks, Apache Spark, Iceberg, and AI technologies. We are looking for a Lead Cloud Data Engineer who is passionate about cloud engineering, Big Data, distributed systems, and modern data platforms.

This role works at the intersection of Cloud, Data Engineering, Analytics, and AI. You will design and deliver scalable, secure, and high-performing data solutions that support advanced analytics, machine learning, and business growth at global scale.

The ideal candidate has hands-on experience with modern cloud technologies as well as large-scale Big Data systems, including Hadoop and Spark, and is equally comfortable architecting cloud-native solutions, developing data pipelines, optimizing distributed workloads, and driving the modernization of enterprise data platforms.

Build modern cloud data solutions. Enable AI and analytics at scale. Shape the future of data-driven marketing.

Role:

Design, develop, and support large-scale cloud-native data solutions on AWS and Databricks.

Build and maintain batch, streaming, and real-time data pipelines supporting analytics, reporting, machine learning, and AI workloads.

Engineer scalable distributed data processing systems using Apache Spark and related Big Data technologies.

Design and implement modern Lakehouse architectures leveraging Databricks, Delta Lake, and Apache Iceberg.

Lead modernization initiatives migrating data workloads from Hadoop-based platforms to cloud-native architectures.

Develop reusable frameworks and engineering patterns that improve scalability, reliability, and operational efficiency.

Partner with Architects, Product Managers, Data Scientists, and Software Engineers to define and implement cloud data solutions.

Evaluate technologies and approaches for data ingestion, transformation, storage, governance, and consumption.

Implement best practices around security, governance, performance optimization, observability, and cost management.

Drive automation using Infrastructure-as-Code, CI/CD pipelines, and modern software engineering practices.

Contribute to the evolution of Marketing Services' cloud and data strategy while enabling future AI and machine learning capabilities.

All About You:

Strong hands-on experience with Big Data technologies including Hadoop, HDFS, Hive, Spark, and related distributed computing frameworks.

Deep expertise in cloud data engineering on AWS, including storage, compute, networking, security, and scalable data architectures.

Experience designing and developing modern data solutions using Databricks, Spark, Delta Lake, and cloud-native technologies.

Proficiency in Python, Java, or Scala, building large-scale distributed data processing applications.

Experience building robust batch and streaming pipelines supporting high-volume and high-velocity workloads.

Understanding of modern Lakehouse architecture principles and open-table formats such as Apache Iceberg and Delta Lake.

Skilled in data modelling, schema design, partitioning strategies, query optimization, and performance tuning.

Familiarity with enterprise-scale data governance, security, privacy, and compliance, working with structured and unstructured datasets at scale.

Experience with Infrastructure-as-Code (e.g., Terraform), modern CI/CD, and cloud-native monitoring and observability practices.

Strong analytical and problem-solving skills, a passion for automation, and the ability to assess emerging technologies and recommend scalable, cost-effective solutions.

Excellent communication and collaboration across technical and business teams; comfortable in Agile/Scrum environments managing multiple priorities.

BS/MS degree in Computer Science, Software Engineering, Information Systems, or a related field.

Preferred Qualifications:

Experience with Databricks on AWS in enterprise production environments.

Experience migrating workloads from Hadoop or Cloudera ecosystems to modern cloud architectures.

Hands-on experience with Apache Kafka, Flink, or other event-streaming technologies.

Experience supporting Machine Learning, MLOps, or AI-driven data platforms.

AWS and/or Databricks certifications.

Familiarity with FinOps and cloud cost optimization practices.

Experience working with petabyte-scale data environments.

Corporate Security Responsibility

All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:

Abide by Mastercard’s security policies and practices;

Ensure the confidentiality and integrity of the information being accessed;

Report any suspected information security violation or breach, and

Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.