Skip to content

Comment on Postgres data stored in Parquet on S3: LTAP architecture explainedparent

Comments

I also want to point out that all of this sounds fun and great — until the load kicks in and usage starts to grow.

That's when you start seeing:

- *Rate limits* from object storage - *Dropped packets* - *Hanging S3 requests* - *Overloaded NVMe drives* — because it turns out they're nowhere near as fast as they seem

For example, we recently discovered that the read speed is 5 GB/s, but the *average write speed is only 400 MB/s* — not several gigabytes as expected. Surprise! Who would have thought that Bare Metal could ship such underwhelming drives?

And then there's the CPU — which is also easy to kill, for instance, if you're compressing chunks. And so on, and so on.

A lot of things surface once you're in *production usage*. On paper, of course, everything looked much simpler.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.