Comment on Postgres data stored in Parquet on S3: LTAP architecture explainedparentComments−nikita2moRecent data plus working set is always in Postgres page format.Historical data when pushed to s3 is in parquet. This happens async - not on the transaction hot path.So older data below certain LSN is on s3 in parquet available to all analytics processing. Hot data is on page servers in page format for OLTP.You can be smart in querying both representations for real time analytical queries−amin22moI assume the parquet files are way larger than the page format, doesn't this cause a lot of read amplification? OLTP side needs to read a lot more data to fetch a single old row that's stored in parquet format.
Comments
Recent data plus working set is always in Postgres page format.
Historical data when pushed to s3 is in parquet. This happens async - not on the transaction hot path.
So older data below certain LSN is on s3 in parquet available to all analytics processing. Hot data is on page servers in page format for OLTP.
You can be smart in querying both representations for real time analytical queries
I assume the parquet files are way larger than the page format, doesn't this cause a lot of read amplification? OLTP side needs to read a lot more data to fetch a single old row that's stored in parquet format.