Short summary: fixes for imports, SCD merge behavior (SCD1 & SCD2), config parsing improvements, schema-evolution handling, Delta identity surrogate-key support, watermark and CDF ingestion fixes. See ...

A production-ready ETL pipeline for ingesting and processing Dutch RDW (Vehicle Authority) car registration data using Databricks, implementing the medallion architecture (Bronze/Silver/Gold) with SCD ...