Upcoming Projects
Detailed walkthroughs of real-world Databricks implementations — from design decisions to production outcomes.
Enterprise Setup
End-to-end implementation of Unity Catalog across multiple workspaces — covering metastore architecture, privilege model design, data lineage tracking, and cross-workspace sharing for a multi-team analytics platform.
Tuning Deep-Dive
Reducing Spark job runtimes by 40%+ through partitioning strategies, adaptive query execution, broadcast join optimisation, and intelligent caching — with before/after benchmarks and Spark UI teardowns.
on Delta Lake
Designing and building a production Medallion (Bronze → Silver → Gold) architecture with Delta Lake — including schema evolution, Z-ordering, VACUUM strategies, and incremental load patterns using Auto Loader.
Pipelines
Building low-latency streaming pipelines with Structured Streaming and Delta Live Tables — ingesting from Azure Event Hubs, applying stateful aggregations, and writing to Gold-layer tables with exactly-once guarantees.
Tools & Technologies
The Databricks ecosystem and complementary Azure services powering these projects.