I tested on the dickens dataset (Charles Dickens novels). This algorithm works well on English text. I do not expect it to be any better than zstd at binary data. I haven't tried code.
It does used zstd. My algorithm is a preprocessing step for zstd.
Comments
What kind of data is it 14% better on?
The algorithm code looks like it's actually using Zstd, is that right?
Please show some benchmarks. I'd like to repeat your test!
I tested on the dickens dataset (Charles Dickens novels). This algorithm works well on English text. I do not expect it to be any better than zstd at binary data. I haven't tried code.
It does used zstd. My algorithm is a preprocessing step for zstd.
Commit that data + test code :)
Add the results to the readme ::))