Clickhouse for storage, airflow for orchestration. Dbt for reusability in your models and transforms. There are a bunch of decent extractors that have prebuilt connectors to pull data from third parties.
Most of all, though, you should think through how users will discover data, and how you will check quality. Once your transforms start getting complicated, you’ll want this (and maybe a metric store).
Since your data isn’t huge, build things so you can rerun in a different architecture, should you outgrow your setup. Keep a certain level of raw data where possible in AWS, make sure your dbt build works from scratch.
While your data size is small, focus on delivering value from as simple an architecture as you can get away with. If you can rebuild your warehouse because you’re using standards, the underlying tech doesn’t matter as much as the processes and culture you establish.
Hmm, I have only ever worked in data warehouses that don't support updates - I think that is a core attribute of a DW. Warehouses are (IME) considered separate from production, with one major difference being that DWs don't need to support updates or transactions.
Comments
Clickhouse for storage, airflow for orchestration. Dbt for reusability in your models and transforms. There are a bunch of decent extractors that have prebuilt connectors to pull data from third parties.
Most of all, though, you should think through how users will discover data, and how you will check quality. Once your transforms start getting complicated, you’ll want this (and maybe a metric store).
Since your data isn’t huge, build things so you can rerun in a different architecture, should you outgrow your setup. Keep a certain level of raw data where possible in AWS, make sure your dbt build works from scratch.
While your data size is small, focus on delivering value from as simple an architecture as you can get away with. If you can rebuild your warehouse because you’re using standards, the underlying tech doesn’t matter as much as the processes and culture you establish.
Clickhouse for storage,
What about deletes? And all that fussy transaction stuff?
It seems strange to make that particular recommendation (for apparently all their storage needs) on the basis of the information provided.
Hmm, I have only ever worked in data warehouses that don't support updates - I think that is a core attribute of a DW. Warehouses are (IME) considered separate from production, with one major difference being that DWs don't need to support updates or transactions.
Deletes are possible in Clickhouse[0]
[0] - https://clickhouse.com/docs/en/sql-reference/statements/alte...
Yes, they're possible -- but it's high-expense operation, intended to be used "infrequently".
Which may or may not impact their use case. Just something to be aware of.