I actually didn't compare gzip and bzip2 before uploading, and I probably should have. I did use `gzip -9`. I'm compressing it with bzip2 -9 now to see if it gets any smaller. I'll post a reply here when I know.
Use xz. Debian, Gnome, Archlinux, Gentoo, Fedora have all switched to xz, and for good reason. I expect xz will offer a 20-30% reduction in file size. That will add up quick.
My download has slowed to a crawl around 90% completion. I am doing 3k a second. When I read that your list was not sorted case sensitively I wondered what difference that may make.
I did a little experiment with the american-wordlist-insane wordlist and sorting in the interim. I used msort which does a case sensitive sort[1] and sort which does not. Here are the results:
Comments
I actually didn't compare gzip and bzip2 before uploading, and I probably should have. I did use `gzip -9`. I'm compressing it with bzip2 -9 now to see if it gets any smaller. I'll post a reply here when I know.
Use xz. Debian, Gnome, Archlinux, Gentoo, Fedora have all switched to xz, and for good reason. I expect xz will offer a 20-30% reduction in file size. That will add up quick.
Thanks, i'll run xz too!
My download has slowed to a crawl around 90% completion. I am doing 3k a second. When I read that your list was not sorted case sensitively I wondered what difference that may make.
I did a little experiment with the american-wordlist-insane wordlist and sorting in the interim. I used msort which does a case sensitive sort[1] and sort which does not. Here are the results:
It is not a big difference but it is still a little better. For future releases you may want to do a case sensitive sort.[1] Zebra comes before aardvark
bzip2 saved about 200mb:
gzip: 4293 bzip2: 4097
xz is still running
For anyone interested, p7zip set for -mx9 got the file down to 2.72GB on my Mac.
I don't know about p7zip but the xz man page cautions against blindly using -9.