While that may be why SpiderOak does what they do, I disagree with most of their arguments?
First, you can do variable block-based de-duplication, which is how major storage vendors do it - not per-file, which doesn't really buy you much.
Leveraging this in the SAN firmware also prevents their ridiculous file transfer 'vulnerability' (which only exists due the way they wrote their software) - all of your files copy over the network to the storage system. Once they're on the storage system, at some later time and asynchronously, the storage system runs a dedupe on the blocks, and winnows down its storage. Think of it like transparently compressing on the storage side, only, hopefully, less intense on I/O.
Finally, they could simply encrypt everything and then they can't answer subpoenas about who has knowledge of what - it was all encrypted immediately after uploading, and no log is kept. If the data doesn't exist, you can't be forced to give it up...
IIRC, SpiderOak doesn't encrypt your files "immediately after uploading". Files are encrypted on the client side even before they're uploaded. Therefore SpiderOak never even sees unencrypted data. That's their "zero knowledge" policy. They think it's better to make it impossible for themselves ever to have any idea what files they're hosting, rather than saying "We used to know X seconds ago, but we don't know anymore."
The blog doesn't mention whether or not SpiderOak uses block-level deduplication. Maybe it's part of their storage infrastructure, maybe isn't. But all that client-side encryption would severely reduce the number of duplicate blocks even if everyone uploaded the same file.
One of the big challenges of scaling a backup service is maximizing the upload rate from customers, so there's a strong incentive to do the de-duplication before the upload when possible. I don't wish to call anyone out specifically, but several storage services operate exactly in this way.
I disagree that requiring end-users to effectively operate their own encryption software is a robust path to privacy.
Comments
While that may be why SpiderOak does what they do, I disagree with most of their arguments?
First, you can do variable block-based de-duplication, which is how major storage vendors do it - not per-file, which doesn't really buy you much.
Leveraging this in the SAN firmware also prevents their ridiculous file transfer 'vulnerability' (which only exists due the way they wrote their software) - all of your files copy over the network to the storage system. Once they're on the storage system, at some later time and asynchronously, the storage system runs a dedupe on the blocks, and winnows down its storage. Think of it like transparently compressing on the storage side, only, hopefully, less intense on I/O.
Finally, they could simply encrypt everything and then they can't answer subpoenas about who has knowledge of what - it was all encrypted immediately after uploading, and no log is kept. If the data doesn't exist, you can't be forced to give it up...
IIRC, SpiderOak doesn't encrypt your files "immediately after uploading". Files are encrypted on the client side even before they're uploaded. Therefore SpiderOak never even sees unencrypted data. That's their "zero knowledge" policy. They think it's better to make it impossible for themselves ever to have any idea what files they're hosting, rather than saying "We used to know X seconds ago, but we don't know anymore."
The blog doesn't mention whether or not SpiderOak uses block-level deduplication. Maybe it's part of their storage infrastructure, maybe isn't. But all that client-side encryption would severely reduce the number of duplicate blocks even if everyone uploaded the same file.
One of the big challenges of scaling a backup service is maximizing the upload rate from customers, so there's a strong incentive to do the de-duplication before the upload when possible. I don't wish to call anyone out specifically, but several storage services operate exactly in this way.
I disagree that requiring end-users to effectively operate their own encryption software is a robust path to privacy.