Well what if you don't get the binary from the same source as the hash, or it's been sitting on a disk for a while and you want to double-check its integrity?
The post to which I replied specifically focused on the case where the binary came from the same source as the hash, and was being used to confirm the integrity of the download. I was not addressing other cases.
(Although I will note that file systems, like application protocols, should maintain their own integrity; however most do not. Which also seems bizarre given that it's 2014.)
Other programs could, accidentally or maliciously, step on the contents of a file. Filesystem checksums are awesome (I use btrfs), but it doesn't protect your file from everything. Hm... maybe a versioned filesystem where you can go back to the "original" version of a file and verify its checksum? :)
Yes, protecting from malicious / accidental modification is a separate problem than protecting from bit rot / gamma rays. The former is obviously much more difficult than the latter. My surprise stems from the fact that common tools are not robust against the latter by now.
(My day job is working with enterprise-grade content-addressed block storage, so maybe I've just set my data-protection expectations too high.)
Indeed, I find it surprising that there's no filesystem flag giving the opposite behaviour to "compress contents to save space": something like "transparently store this file bloated by a fountain codec to ensure integrity on an bitwise-unreliable backing store" or maybe "transparently generate a Parity Volume Set from this file and stripe it across the disk to protect against lost sectors."
Both of these options would protect your important files against failing single disks without having to do any RAIDing. The unimportant data (e.g. the OS itself, caches, etc.) could be reduced-redundancy, since it doesn't need to be captured in a disk rescue.
It would be a great feature, but obviously adds complexity. You can get half way there with single disk ZFS, which lets you store multiple copies. But the key feature is that files and directory info are checksummed, and regularly checked, so corruption cannot silently occur. But I think that usually the whole disk is lost, rather than getting bad blocks (so 90's).
Comments
Well what if you don't get the binary from the same source as the hash, or it's been sitting on a disk for a while and you want to double-check its integrity?
The post to which I replied specifically focused on the case where the binary came from the same source as the hash, and was being used to confirm the integrity of the download. I was not addressing other cases.
(Although I will note that file systems, like application protocols, should maintain their own integrity; however most do not. Which also seems bizarre given that it's 2014.)
Other programs could, accidentally or maliciously, step on the contents of a file. Filesystem checksums are awesome (I use btrfs), but it doesn't protect your file from everything. Hm... maybe a versioned filesystem where you can go back to the "original" version of a file and verify its checksum? :)
Yes, protecting from malicious / accidental modification is a separate problem than protecting from bit rot / gamma rays. The former is obviously much more difficult than the latter. My surprise stems from the fact that common tools are not robust against the latter by now.
(My day job is working with enterprise-grade content-addressed block storage, so maybe I've just set my data-protection expectations too high.)
Indeed, I find it surprising that there's no filesystem flag giving the opposite behaviour to "compress contents to save space": something like "transparently store this file bloated by a fountain codec to ensure integrity on an bitwise-unreliable backing store" or maybe "transparently generate a Parity Volume Set from this file and stripe it across the disk to protect against lost sectors."
Both of these options would protect your important files against failing single disks without having to do any RAIDing. The unimportant data (e.g. the OS itself, caches, etc.) could be reduced-redundancy, since it doesn't need to be captured in a disk rescue.
It would be a great feature, but obviously adds complexity. You can get half way there with single disk ZFS, which lets you store multiple copies. But the key feature is that files and directory info are checksummed, and regularly checked, so corruption cannot silently occur. But I think that usually the whole disk is lost, rather than getting bad blocks (so 90's).
For any archival DVDs I burn, I compress, then run through PAR2 http://en.m.wikipedia.org/wiki/Parchive