Skip to content

Comment on Databricks open-sources Delta Lake to make data lakes more reliableparent

Comments

Practical definitions:

Data lake = place to store unstructured raw data. Usually as files in an object store or Hadoop/HDFS cluster. Analyse with data processing/SQL frameworks. Schema may be part of the data (parquet, avro) or on-read (raw csvs or json modeled into tables).

Data warehouse = place to store processed (semi)structured data. Usually in a distributed columnstore database. Analyze with SQL. Schema is pre-defined by the tables. Usually for smaller, faster, real-time queries or as a cache in-front of a data lake.

Some cloud data warehouses like BigQuery and Snowflake can also query unstructured files and even run on top of the object store so the boundaries are getting blurred. Will probably converge at some point in the future.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.