> It's not architecturally impossible for us to support long-running read snapshots, but it is expensive for our storage servers to keep snapshots alive for a long time. So our backup solution instead works by backing up transaction logs while doing a non-isolated read of the database. At restore time we will replay the logs to get back a consistent point-in-time snapshot.
Sure, it's all technically possible -- Postgres will simply stop collecting garbage via VACUUM, for example -- but the results are not usually very pleasant. But it sounds like instead your solution to the snapshot-read is more like how the online binary backups work, whereby you combine transaction logs with inconsistent reads. That system works very well. On the other hand, "logical" backups have been a major problem for me; sidestepping this by using lower level transaction log mechanics and dirty reads I think is a much better idea.
> The reason that we see higher burst than steady state performance has nothing to do with transactions. We have to do reads from SSD as the application requests them, and we have to make writes durable by logging them right away, but as you surmise we can defer the hard work of doing random writes to our btrees for a while. Even a workload that is 90% reads benefits a lot from deferring writes because writes have to be 3x replicated and because consumer SSDs are comparatively slow at mixed read/write workloads. Read only workloads will not see any initial burst (and might see a ramp-up time from cold cache effects).
I see. It's not read-only, it's 90/10, I misread. That makes more sense. What's your read-only performance? Presumably it would be impacted by requiring fencing to deal with cases involving network partitions, which is one of the problem with non-partitioned and consistent systems systems, as far as I can tell: reads need to check for liveness, and that's much harder than....doing nothing.
Random, uncached reads on the same dataset as the 90/10 - 1.6M/sec, limited by SSD throughput. Cached reads (from the same dataset, with a different distribution of read keys) you can see on our website at 3.2 M/sec. I should mention that the test clients also run on the same cluster, and in that workload they are a significant fraction of CPU.
Read requests come to a storage server with a specific version number attached, so the storage server can reply authoritatively without any further communications. Starting a transaction requires selecting a version number for the read snapshot, and that incurs some latency to deal with the concerns you mention (but scales fine).
Comments
> It's not architecturally impossible for us to support long-running read snapshots, but it is expensive for our storage servers to keep snapshots alive for a long time. So our backup solution instead works by backing up transaction logs while doing a non-isolated read of the database. At restore time we will replay the logs to get back a consistent point-in-time snapshot.
Sure, it's all technically possible -- Postgres will simply stop collecting garbage via VACUUM, for example -- but the results are not usually very pleasant. But it sounds like instead your solution to the snapshot-read is more like how the online binary backups work, whereby you combine transaction logs with inconsistent reads. That system works very well. On the other hand, "logical" backups have been a major problem for me; sidestepping this by using lower level transaction log mechanics and dirty reads I think is a much better idea.
> The reason that we see higher burst than steady state performance has nothing to do with transactions. We have to do reads from SSD as the application requests them, and we have to make writes durable by logging them right away, but as you surmise we can defer the hard work of doing random writes to our btrees for a while. Even a workload that is 90% reads benefits a lot from deferring writes because writes have to be 3x replicated and because consumer SSDs are comparatively slow at mixed read/write workloads. Read only workloads will not see any initial burst (and might see a ramp-up time from cold cache effects).
I see. It's not read-only, it's 90/10, I misread. That makes more sense. What's your read-only performance? Presumably it would be impacted by requiring fencing to deal with cases involving network partitions, which is one of the problem with non-partitioned and consistent systems systems, as far as I can tell: reads need to check for liveness, and that's much harder than....doing nothing.
Random, uncached reads on the same dataset as the 90/10 - 1.6M/sec, limited by SSD throughput. Cached reads (from the same dataset, with a different distribution of read keys) you can see on our website at 3.2 M/sec. I should mention that the test clients also run on the same cluster, and in that workload they are a significant fraction of CPU.
Read requests come to a storage server with a specific version number attached, so the storage server can reply authoritatively without any further communications. Starting a transaction requires selecting a version number for the read snapshot, and that incurs some latency to deal with the concerns you mention (but scales fine).