GCP Data Engineer

Description

The company is a global partner for AI-first digital transformation and engineering, with more than 25 years in the field, around 5,000 staff across 16 countries, and a focus on scalable products, platforms and data-driven solutions.

Its work covers artificial intelligence, data and cloud, intelligent automation and digital products.

The CEE team has close to 1,000 specialists building applications, data platforms and AI solutions for clients, using AWS, Azure, GCP, Databricks and Snowflake, with a consulting approach and an emphasis on knowledge sharing and growth.

Responsibilities

• Build, run and tune ETL and ELT pipelines on Google Cloud Platform.
• Tune Spark jobs running on Dataproc and Dataproc Serverless.
• Create routing logic for workloads and mechanisms that keep cloud costs down.
• Set up and maintain data ingestion for the Bronze, Silver and Gold layers.
• Structure and manage raw data in Google Cloud Storage.
• Build event-driven orchestration with Pub/Sub and Apache Airflow.
• Write, maintain and improve Airflow DAGs and the orchestration frameworks around them.
• Add support for reprocessing and backfilling data.
• Merge related processing workflows into reusable orchestration patterns.
• Extend and improve metadata-driven generators and platform tooling.
• Support the generation, regeneration and distribution of artifacts.
• Improve monitoring, reliability and day-to-day support for data pipelines.
• Work with platform, analytics and DevOps teams on architecture and implementation choices.

Requirements

• Solid commercial experience building and running data platforms on Google Cloud Platform.
• Hands-on work with Dataproc and Apache Spark.
• Experience designing and building ETL and ELT pipelines.
• Experience with BigQuery and Google Cloud Storage.
• Strong SQL and Python skills.
• Experience improving data processing performance and cloud spend.
• Experience designing ingestion frameworks and reusable data processing components.
• Knowledge of data platform architecture patterns, including Bronze, Silver and Gold layers.
• Experience troubleshooting and supporting production data pipelines.
• Strong hands-on experience with Apache Airflow.
• Experience designing, implementing and maintaining complex DAGs.

Conditions

You must work from the European Union and hold a work permit.

Hiring has five steps: CV review, HR call, interview, client interview, then decision.