About Experience Skills Certs Databricks Fabric Hire Me
Portfolio · Databricks

Databricks
Engineering

Case studies on Unity Catalog, Delta Lake architectures, PySpark performance tuning, and end-to-end lakehouse implementations. Deep-dive technical walkthroughs — coming soon.

Work in Progress
Databricks Certified
Case Studies

Real-World Implementations

End-to-end Databricks solutions built from actual industry scenarios — covering ingestion, transformation, security, and governance.

GreenGrid Solutions — Global Energy and Grid Technology
GreenGrid Solutions · Global Energy and Grid Technology · Thermal Utility · 100 Home Pilot
BRONZE meter_readings.json customer_pii.csv Raw Ingest TRANSFORM (PySpark) A · Quality Filter — kwh > 0 AND < 100 B · SHA-256 Email Hash (no raw PII) C · Peak Pricing — when(18–21h) × 2 SILVER Silver_SmartBill Quarantine_Readings Clean Output Bad Rows → Quarantine GREENGRID SOLUTIONS · PIPELINE ARCHITECTURE Bronze → Transform → Silver · Medallion Pattern
01 · Industry Context
Completed W1-CS-01

GreenGrid Solutions
Smart Pricing Pipeline

A 100-home smart grid pilot needed dynamic billing (peak vs. off-peak rates), real-time data quality enforcement, and email PII anonymisation — all in a single automated Databricks pipeline replacing a broken Excel process.

A
Quality Filter
Bad rows (negatives, 9000 kWh spikes) → Quarantine_Readings
B
PII Security Hash
SHA-256 email anonymisation — no raw PII in Silver layer
C
Peak Hour Pricing
6–9 PM → BasePricePerKwh × 2 · Bill = kwh × FinalRate
PySpark Delta Lake SHA-256 Medallion Data Quality when/otherwise
SantéFlux — Health-Tech & Data Solutions, Paris
SantéFlux · Health-Tech & Data Solutions · Paris, France · GDPR Privacy Crisis
SANTEFLUX · GDPR PRIVACY CRISIS · PARIS HEALTH-TECH ♥ Bio-Pings Millions/day Smart Watches PRIVACY SHIELD PIPELINE ① PII Mask — regexp_replace(Full_Name) ② sha2(concat(User_ID, salt), 256) ③ Drop FROZEN sensor rows ④ groupBy(City, Sub_Type).agg(avg) ⚠ Exchange hashpartitioning → SHUFFLE WALL City Stress Matrix City SubType AvgHR Paris Premium 88.4 Lyon Free 92.1 Marseille Premium 85.7 GDPR Compliant Output ⚠ CPO Moreau Analytics Access Blocked
SANTEFLUX · SPARK PHYSICAL PLAN · BROADCAST JOIN STRATEGY Vitals Table 750k rows · ~43 MB Secure_ID · City · Heart_Rate Filter: HR > 0 ← PUSHED DOWN ✓ Ghost Users → left_anti join Subscription Table 10,000 rows · ~125 KB → broadcast() candidate Abonnement Premium / Free BroadcastHashJoin Inner join on Secure_ID Tungsten off-heap memory management ⚠ SHUFFLE WALL Exchange hashpartitioning(City, 200) groupBy + pivot → high network cost HashAggregate avg(Heart_Rate) by City + Sub_Type → City Stress Matrix (VIP analysis) dbutils.widgets Board toggles City without re-running pipeline Raw: 750k rows · 85 MB test set Output: GDPR-safe City Stress Matrix
01 · Industry Context
Completed W2-CS-01

SantéFlux
The GDPR Privacy Crisis

A Paris-based health-tech company processing millions of daily bio-pings from smart watches hit a hard blocker: raw patient PII was exposed in shared analytics folders, triggering a CPO-ordered analytics lockdown days before a critical board meeting. The business needed a GDPR-compliant "Privacy Shield" pipeline — masking names, salting & hashing User IDs, purging frozen sensor rows, and delivering a City Stress Matrix showing average heart rates by location and subscription tier, all without compromising Spark performance under a 750k-row load.

🔒
PII Masking & Salted Hashing
regexp_replace() scrubs Full_Name at ingestion · sha2(concat(User_ID, salt), 256) stops cross-team re-identification
⚡
Shuffle Wall & Predicate Pushdown
Frozen sensor rows dropped early via pushed-down Filter · pivot() cost explained via .explain() · Shuffle cost reduced before aggregation
📡
Broadcast Join & Ghost Users
10k-row Subscription table broadcast() to all workers · left_anti join isolates Ghost Users · dbutils.widgets lets the board toggle cities live
GDPR Compliance sha2 · concat Salted Hashing Broadcast Join Predicate Pushdown Shuffle pivot · groupBy dbutils.widgets left_anti Tungsten Engine
! BONDI MANLY COOGEE CRONULLA 04:00 AM · SYDNEY ● LIVE PATROL | DRONES ACTIVE: 4 | ⚠ DATA CORRUPTION DETECTED | BlueGuard v3.1
AeroSurf NSW · Autonomous Beach Patrol · Sydney, Australia · Delta Lake Emergency Recovery
DELTA LAKE MEDALLION ARCHITECTURE · BLUEGUARD MODEL-V D-101, D-102, D-103 MODEL-X X-500, X-501 (+ lidar) BRONZE Bronze_Drone_Log mergeSchema=true CDF Enabled 7-yr retention SCD 1+2 SILVER Sector_Status (SCD2) risk_level → Type 2 history battery/temp → Type 1 update lidar_depth NULL → -1.0 is_current · start/end_date OPTIMIZE GOLD Morning_Safety_Brief CDF Audit Report VACUUM optimised ∞ retention 6 AM BRIEF Lifeguards DELTA LAKE RECOVERY TOOLKIT ⏪ Time Travel 🔀 Shallow Clone 🔁 RESTORE TABLE 📋 CDF Audit
INCIDENT TIMELINE · 04:00–06:00 AM 03:45 Sarah's Email Dashboard shows ZERO sharks. 40 beaches at risk. 03:52 Legal Mandate Marcus Li demands forensic audit trail. Chain of custody req. 04:15 💥 THE SABOTAGE Junior dev UPDATE without WHERE clause. All sharks → 'Dolphin'. (Recovered via Time Travel) 04:30 RESTORE TABLE Bronze reverted to safe version. CDF audit CSV ready. 06:00 ✅ BEACHES OPEN Morning Safety Brief live. Lifeguards cleared all sectors. SCHEMA EVOLUTION · mergeSchema — Model-V (5 cols) + Model-X (7 cols) unified in Bronze without pipeline crash SCD Type 2 MERGE · Shallow Clone Rescue_Clone · DESCRIBE HISTORY forensics · OPTIMIZE + VACUUM storage report · CDF legal CSV ✓ RESOLVED | 10 DELIVERABLES · DELTA LAKE · SCD 1+2 · TIME TRAVEL · SHALLOW CLONE · CDF · OPTIMIZE · VACUUM
01 · Night Patrol Scene
Completed W3-CS-01

AeroSurf NSW
Project BlueGuard

It's 4 AM in Sydney. A junior developer's missing WHERE clause just renamed every historical shark sighting to "Dolphin" — the Morning Safety Brief now shows zero sharks, and 40 beaches hang in the balance. As Lead Data Engineer for AeroSurf NSW, you must ingest a schema-drifting fleet of drone batches (Model-V + Model-X), implement a Hybrid SCD Type 1/2 MERGE, execute a Delta Lake Time Travel rescue, satisfy a legal forensic audit via Change Data Feed, and deliver a certified OPTIMIZE + VACUUM storage report — all before the 6 AM lifeguard briefing.

🛸
Schema Evolution & mergeSchema
Model-V (5 cols) + Model-X (7 cols, +water_temp, +lidar_depth) unified into Bronze_Drone_Log without crashing the pipeline
🔀
Hybrid SCD MERGE (Type 1 + Type 2)
risk_level change → expire old row + insert new history · battery/temp change → in-place UPDATE only · lidar_depth NULL → default -1.0
⏪
Time Travel Recovery & Legal Audit
DESCRIBE HISTORY forensics → Shallow Clone (Rescue_Clone) → RESTORE TABLE · CDF CSV chain-of-custody · OPTIMIZE + VACUUM storage metrics
Delta Lake mergeSchema SCD Type 1+2 Time Travel Shallow Clone RESTORE TABLE Change Data Feed DESCRIBE HISTORY OPTIMIZE · VACUUM Auto Loader GDPR · PII Handling
Case Studies

More Case Studies Brewing

Detailed walkthroughs of real-world Databricks implementations — from design decisions to production outcomes.

Tech Stack

Tools & Technologies

The Databricks ecosystem and complementary Azure services powering these projects.

DB
Azure Databricks
Runtime · Clusters · Jobs
UC
Unity Catalog
Governance · Lineage
Δ
Delta Lake
ACID · Time Travel
⚡
Apache Spark
PySpark · SQL · Streaming
ADL
ADLS Gen2
Storage · Raw Landing
ADF
Azure Data Factory
Orchestration · ETL