I am downloading the file now. I was a little surprised to see that you went with gzip. Given the target audience it is not that hard to imagine that the end user will have access to bzip2 or xz. With the size of the file why choose gzip? Hopefully `gzip -9` was used. I am curious about how much smaller xz/bzip2 will be. I will update this post once the file is downloaded.
I actually didn't compare gzip and bzip2 before uploading, and I probably should have. I did use `gzip -9`. I'm compressing it with bzip2 -9 now to see if it gets any smaller. I'll post a reply here when I know.
Use xz. Debian, Gnome, Archlinux, Gentoo, Fedora have all switched to xz, and for good reason. I expect xz will offer a 20-30% reduction in file size. That will add up quick.
My download has slowed to a crawl around 90% completion. I am doing 3k a second. When I read that your list was not sorted case sensitively I wondered what difference that may make.
I did a little experiment with the american-wordlist-insane wordlist and sorting in the interim. I used msort which does a case sensitive sort[1] and sort which does not. Here are the results:
Comments
I am downloading the file now. I was a little surprised to see that you went with gzip. Given the target audience it is not that hard to imagine that the end user will have access to bzip2 or xz. With the size of the file why choose gzip? Hopefully `gzip -9` was used. I am curious about how much smaller xz/bzip2 will be. I will update this post once the file is downloaded.
I actually didn't compare gzip and bzip2 before uploading, and I probably should have. I did use `gzip -9`. I'm compressing it with bzip2 -9 now to see if it gets any smaller. I'll post a reply here when I know.
Use xz. Debian, Gnome, Archlinux, Gentoo, Fedora have all switched to xz, and for good reason. I expect xz will offer a 20-30% reduction in file size. That will add up quick.
Thanks, i'll run xz too!
My download has slowed to a crawl around 90% completion. I am doing 3k a second. When I read that your list was not sorted case sensitively I wondered what difference that may make.
I did a little experiment with the american-wordlist-insane wordlist and sorting in the interim. I used msort which does a case sensitive sort[1] and sort which does not. Here are the results:
It is not a big difference but it is still a little better. For future releases you may want to do a case sensitive sort.[1] Zebra comes before aardvark
bzip2 saved about 200mb:
gzip: 4293 bzip2: 4097
xz is still running
For anyone interested, p7zip set for -mx9 got the file down to 2.72GB on my Mac.
I don't know about p7zip but the xz man page cautions against blindly using -9.