Comment on We could save petabytes of cache storage with Zstandard and PingoraparentComments−colechristensen9dYou may or may not be familiar with the content defined chunking family of algorithms like FastCDC.https://joshleeb.com/posts/chunking.htmlhttps://www.usenix.org/conference/atc16/technical-sessions/p...On the side I'm working on an extension to git-lfs to use fastcdc for both storage and transmission of artifacts to drastically reduce size and make git-lfs more practical for more things.
Comments
You may or may not be familiar with the content defined chunking family of algorithms like FastCDC.
https://joshleeb.com/posts/chunking.html
https://www.usenix.org/conference/atc16/technical-sessions/p...
On the side I'm working on an extension to git-lfs to use fastcdc for both storage and transmission of artifacts to drastically reduce size and make git-lfs more practical for more things.