Senior Data Platform Engineer (Microsoft Fabric)

Description

The client is building a new risk analysis and scoring platform on Microsoft Fabric. This role owns the medallion data architecture from end to end, covering Bronze, Silver and Gold layers.

This is the most heavily loaded engineering role on the project. It supports both Phase 1 (platform foundation) and Phase 2 (risk product delivery), and you will work with a Senior Technical Lead, a Senior DevOps engineer and a mid-level developer.

The platform takes in transaction and merchant data over SFTP and API. Sensitive PAN data is tokenized at ingestion using the reserved-BIN HMAC-derived format-preserving scheme. Data lands in canonical Silver on Microsoft Fabric, following the company's internal canonical spec.

You will build the MERGE pipeline that fills canonical Silver, expose it through OneLake shortcuts, and produce risk-specific Gold tables. These feed custom DMN rules and pre-scored Mastercard Brighterion feeds.

Responsibilities

• Design and build Bronze, canonical Silver and Gold Delta table schemas with PAN-safe physical layout, including statistics allowlists, liquid clustering, retention settings and V-Order optimization.
• Write Fabric Spark notebooks for the feature pipeline: rolling velocity features over 24h, 7d and 30d windows, merchant-level rolling features, hierarchical rollups, and cross-merchant features joined on pantoken.
• Write a separate Silver adapter for each flavor version, one function per flavor, combined by name, and test each against golden fixtures.
• Build the canonical MERGE DAG from Bronze to Silver, keyed on naturalkey and idempotent under Aperia's canonical Fabric internal spec v1.4.
• Build the Gold materialization pipeline for facttransactionrisk and factmerchantrisk, with correct score-source stamping and idempotent merge behaviour.
• Design Direct Lake semantic models for each reporting surface and performance-test them against representative data volumes.
• Set up OneLake namespace layouts, workspace catalogs and cross-workspace shortcuts with the residuals programme.
• Create idempotent Delta write patterns from Python containers using delta-rs: segmented parallel writes that finish in one atomic commit.
• Size Fabric capacity: choose F-SKUs, run load tests, and decide between reservations and pay-as-you-go.
• Work with the DevOps engineer on Fabric infrastructure as code, covering Terraform capacity provisioning and Fabric REST workspace bootstrap.
• Advise on a ClickHouse fallback if Direct Lake latency is not good enough for client-facing UIs.

Requirements

• At least 5 years in data engineering, including at least 2 years of hands-on Spark development, PySpark preferred.
• Deep knowledge of Delta Lake internals: transaction log format, checkpoints, retention settings (logRetentionDuration and deletedFileRetentionDuration), VACUUM, OPTIMIZE, deletion vectors and statistics collection.
• Strong Parquet expertise: footers, statistics allowlists, row-group sizing, dictionary encoding and column indexes.
• Hands-on Microsoft Fabric experience with workspaces, capacities, notebooks, lakehouses, semantic models, Direct Lake and the SQL analytics endpoint.
• Skill in SQL performance tuning for both OLAP (Fabric) and OLTP (PostgreSQL) workloads.
• Solid Python data engineering skills with pandas, PyArrow and delta-rs.
• Able to explain architecture clearly in writing and to work independently on well-defined features.
• Nice to have: experience on a PCI-scoped data platform.
• Nice to have: background in financial services or payments.
• Nice to have: Power BI semantic model authoring with DAX.
• Nice to have: familiarity with published data-standard governance patterns, such as logical model versus physical profile, SCD2 dimensions, natural-key idempotency and extension attribute stores.
• Nice to have: OneLake shortcuts and cross-workspace lakehouse patterns.
• Nice to have: ClickHouse or a similar OLAP engine.
• Nice to have: Terraform for Azure data resources.
• Nice to have: prior work with fluid or schema-flexible ingestion patterns.
• Additional information is listed in the original posting.

Conditions

Remote or flexible cooperation is available. Travel is involved in some international projects.

Pay, benefits and perks may vary depending on the form of cooperation you choose.

The company is an equal-opportunity employer.

Benefits

• International projects with leading global clients.
• A non-corporate working atmosphere.
• Language classes.
• Access to knowledge-sharing initiatives.
• Private healthcare and life insurance.
• A multisport card.