On the surface (I haven't tested it) it looks like a great cost and runtime saver, but only for data sets that need a cadence above 5 or so minutes. You wouldn't be able to have a refresh button to get the "latest" data outside this window, depending on size and build time? Wondering if you can apply a hybrid approach, combining the historical parquet file with a live query.
Comments
On the surface (I haven't tested it) it looks like a great cost and runtime saver, but only for data sets that need a cadence above 5 or so minutes. You wouldn't be able to have a refresh button to get the "latest" data outside this window, depending on size and build time? Wondering if you can apply a hybrid approach, combining the historical parquet file with a live query.
Cloudflare Pipelines has a configurable interval to write to parquet files that's as low as 10s: https://developers.cloudflare.com/pipelines/sinks/available-... so you could have a pretty fast refresh
It will cost (slightly) more to write this frequently to R2 since you are charged per-write, but this is something you can tune.