Comment on Text classification with Python 3.14's ZSTD moduleparentComments−nl6moGiven that LLMs are just lossless compression machines, I do sometimes wonder how much better they are at compressing plain text compared to zstd or similar. Should be easy to calculate...The current leader on the Hutter Prize (http://prize.hutter1.net/) are all LLM based.It can (slowly!!) compress a 1GB dump of Wikipedia to 106MbBy comparison GZip can compress it to 321MbSee https://mattmahoney.net/dc/text.html for the current leaderboard
Comments
The current leader on the Hutter Prize (http://prize.hutter1.net/) are all LLM based.
It can (slowly!!) compress a 1GB dump of Wikipedia to 106Mb
By comparison GZip can compress it to 321Mb
See https://mattmahoney.net/dc/text.html for the current leaderboard