There are a few features missing from the FOSS Scala/Spark implementation of Delta Lake, but I wouldn't say a lot. The FOSS version supports all the table features in the Delta Lake protocol.
The Delta Rust implementation is missing more table features, but we're closing the gap fast. We just added support for constraints to Delta Rust and are working on change data feed right now.
I’d take issue with the “Iceberg is slow” theme that Databricks in particular has tried to push.
If that were true, Snowflake would not be as fast on Iceberg/Parquet as its native format. The engine makes something fast or slow, not the table format.
Back when were choosing between the three formats about 1.5 years ago, Iceberg was definitely the slowest. If the situation has changed since then, I would love to see an updated comparison.
We tested all three of them using Spark batches that converted a stream of changes into SCD2.
Comments
Delta is nice, but a lot of features are missing from the FOSS version.
Hudi is nice, but they are in the middle of a big format change right now.
Iceberg is nice, but is the most conservative and slow format out of three.
There are a few features missing from the FOSS Scala/Spark implementation of Delta Lake, but I wouldn't say a lot. The FOSS version supports all the table features in the Delta Lake protocol.
The Delta Rust implementation is missing more table features, but we're closing the gap fast. We just added support for constraints to Delta Rust and are working on change data feed right now.
Delta Live Tables and automatic vacuuming are the two big features I'm missing.
Delta Live Tables are a Databricks feature and aren't related to Delta Lake.
Can't you just setup a cron job to vacuum periodically?
DLT is a big feature of a full platform, you can’t really say that’s a missing feature of a delta library.
I’d take issue with the “Iceberg is slow” theme that Databricks in particular has tried to push.
If that were true, Snowflake would not be as fast on Iceberg/Parquet as its native format. The engine makes something fast or slow, not the table format.
Disclaimer - am at Snowflake.
Back when were choosing between the three formats about 1.5 years ago, Iceberg was definitely the slowest. If the situation has changed since then, I would love to see an updated comparison.
We tested all three of them using Spark batches that converted a stream of changes into SCD2.