What's the difference between a Delta Lake and Change Data Capture? Seems like in both cases you're creating a type 2 dimension against a source table.
It is a different technology entirely. CDC is just the log of changes on a relational table. Delta Lake appears to be providing more native administrative capabilities to a data lake implementation (schemas, transactions, cataloging).
CDC you are looking at the change logs of a Relational Database typically.
With a (batch) Data Lake you accept as input different file types (JSON, CSV, AVRO) from various systems. Could be Hadoop systems, could be from a COBOL system on a mainframe.
The Delta here isn’t so related to CDC. This is a replacement for the ad how methods people use now to keep multiple readers and writers of datasets in parquet format in agreement when using Spark’s dataframe API.
Comments
What's the difference between a Delta Lake and Change Data Capture? Seems like in both cases you're creating a type 2 dimension against a source table.
It is a different technology entirely. CDC is just the log of changes on a relational table. Delta Lake appears to be providing more native administrative capabilities to a data lake implementation (schemas, transactions, cataloging).
CDC you are looking at the change logs of a Relational Database typically.
With a (batch) Data Lake you accept as input different file types (JSON, CSV, AVRO) from various systems. Could be Hadoop systems, could be from a COBOL system on a mainframe.
The Delta here isn’t so related to CDC. This is a replacement for the ad how methods people use now to keep multiple readers and writers of datasets in parquet format in agreement when using Spark’s dataframe API.