Slicehost (now under Rackspace) sent a very cryptic email a few weeks ago saying users needed to "migrate" their servers, but gave no reason why. Then this mail went out on Friday:
We are writing to update you on the migration you recently completed, and to provide more information about the reasons it was necessary. We know that migrations can be inconvenient, and we thank you for your patience. Now that the migrations are complete, there is nothing more that you need to do regarding this issue.
When we announced the recent migrations, we explained that such measures are periodically required to promote the stability, performance, security, and feature-richness of our Slicehost platform. We were not able to share more information at the time, without putting you and other customers at risk. Now that the migrations have been completed, however, we want to provide you with the transparency that you expect from Slicehost. We now can tell you the timing of the migrations was driven by the need to fix a potential security issue.
We discovered the issue in collaboration with an independent I.T. security consulting firm that tested our products. The security consultants used forensic techniques to examine the underlying physical disk in newly created instances. They discovered that, in certain use cases, random fragments of temporarily stored data could be left behind on the physical disk.
In repairing this vulnerability, we have ensured that all data is wiped effectively whenever a customer vacates hard-drive space on a host machine. And through the migration that you and other customers have completed, we have cleaned up all fragments of remnant data. The security consulting firm that discovered this issue has performed follow-up testing and has found no remnant data on our environment.
We know of no case of customer data being seen or exploited in any way by any unauthorized party.
One reason is that the remnant data could not have been seen through normal use of slices, but would have had to be sought, using forensic techniques. It was not possible for anyone to specifically target a particular customer through this vulnerability, given the random and fragmented nature of the remnant data. Customers who encrypted sensitive data on slices would have faced no risk of exposure.
If we had made this issue public earlier, we could have opened the door for a malicious user to exploit the vulnerability. For that reason, we decided to keep information about the vulnerability within our company — until now, when the issue has been fully resolved.
Dealing with security issues is a constant in any type of computing or cloud-hosting provider. At Slicehost, we work to provide you with the safest, most-stable environment possible. We regularly consult with independent security consultants. We employ a large and growing staff of security specialists and IT engineers. We are proud of their work in repairing this vulnerability, and grateful for your patience.
If you have questions, please reach out to your support team. We are here to serve you.
The problem: When provisioning a disk, the provider reuses an old disk that has old customer data on it. Part of the disk is overwritten with the OS image but the rest is not.
The fix: Zero out the disk. Either when de-provisioning it, or when provisioning it.
The author's ability to turn 49 words into 2,543 words is quite impressive!
One thing to consider here is keeping sensitive information on virtualized hosting setups block-level (full-disk) encrypted, for instance with Truecrypt.
Obviously it might seem silly to make this comment here, but would this really protect in many cases besides this one? It seems like almost every time the intrusion would occur somewhere where the intruder will have access to the mounted encrypted disk, no?
The area on the SAN occupied by your disk contained a simple disk image. When your image is deleted and another is created overlapping the same spot on the host storage, they can look at their free space to see the contents of your disk.
If your disk happened to be encrypted, their free space would appear to contain garbage data.
Encrypting your disk would do you little good versus your provider if your VM can boot without your intervention... someone could just boot your VM to look inside.
A very interesting article about data leakage in cloud resource allocation. I'm impressed by the steps taken by all parties involved. Security vulnerabilities will continue to creep up over time as they do in most systems/services and the seriousness with which providers treat them is ever more important. What we have to remember is it's not just individuals hosting their own content that are at risk. More and more startups are utilizing cloud infrastructure which means we as consumers must be aware of where our personal data is going. I would love to see some sort of auditory process in place that verifies the security of cloud providers regularly and we as consumers having access to this information.
We had need for short-term (several hours) EC2 compute nodes with sensitive data from customers. They'd boot up and encrypt all the drives using a random key without persisting it anywhere. Even so much as rebooting them made any data unusable. YMMV =) I'm thinking one could set the key using a start-up file so reboot would be possible.
This is not just limited to "cloud" services. Hosting companies recycle hard drives all the time. If you're lucky, they'll wipe it after they pull it out of your machine. If you're not, they won't.
It all comes down to what kind of recycling policies they have for old hardware. Ideally, all drives should be wiped and then stress-tested to avoid the problem where you eventually wind up with nothing but bad disks in the recycle loop. After all, good disks stay in production and thus exit the loop, whereas bad disks fail and come right back.
This isn't "new" information, and many providers already wipe before reprovisioning space. Off the top of my head, I'm aware of three. The paper's sample size is frightfully small (VPS.NET, in particular, is not a shining star of cloud hosting). All you have to do to find out if this is the case is run this on new storage that you've requested from your provider:
strings /dev/<whatever>
This has nothing to do with "cloud", aside from rapidly reusing disks in that environment. This is the same problem with reusing disks in a datacenter, reusing disks at a dedicated host, reusing storage even inside a corporation, and so on. There is nothing "cloud-specific" about this in the slightest.
Of course, they wait until near the bottom of the paper to tell you that, so the title of the paper seems like a needless cloud scare which is really a shared-environment problem. Clueful people in hosting figured this out a while ago (publicly-used hosting has been around a long, long time). Long enough ago that you can measure how clueful your provider is by the usefulness of this trick.
The cloud angle is important because, as a cloud user, I have no control over the situation. More so, most people just expect the provider to take care of them as a result.
Regardless, it's why I'm willing to pay a little more for a company that will be proactive.
At first I thought this was going to be the issue where people were sharing their EC2 instances and were leaving their keys or un-wiped copies of their private keys or tokens on images that others than stumble upon.
Comments
Slicehost (now under Rackspace) sent a very cryptic email a few weeks ago saying users needed to "migrate" their servers, but gave no reason why. Then this mail went out on Friday:
We are writing to update you on the migration you recently completed, and to provide more information about the reasons it was necessary. We know that migrations can be inconvenient, and we thank you for your patience. Now that the migrations are complete, there is nothing more that you need to do regarding this issue.
When we announced the recent migrations, we explained that such measures are periodically required to promote the stability, performance, security, and feature-richness of our Slicehost platform. We were not able to share more information at the time, without putting you and other customers at risk. Now that the migrations have been completed, however, we want to provide you with the transparency that you expect from Slicehost. We now can tell you the timing of the migrations was driven by the need to fix a potential security issue.
We discovered the issue in collaboration with an independent I.T. security consulting firm that tested our products. The security consultants used forensic techniques to examine the underlying physical disk in newly created instances. They discovered that, in certain use cases, random fragments of temporarily stored data could be left behind on the physical disk.
In repairing this vulnerability, we have ensured that all data is wiped effectively whenever a customer vacates hard-drive space on a host machine. And through the migration that you and other customers have completed, we have cleaned up all fragments of remnant data. The security consulting firm that discovered this issue has performed follow-up testing and has found no remnant data on our environment.
We know of no case of customer data being seen or exploited in any way by any unauthorized party.
One reason is that the remnant data could not have been seen through normal use of slices, but would have had to be sought, using forensic techniques. It was not possible for anyone to specifically target a particular customer through this vulnerability, given the random and fragmented nature of the remnant data. Customers who encrypted sensitive data on slices would have faced no risk of exposure.
If we had made this issue public earlier, we could have opened the door for a malicious user to exploit the vulnerability. For that reason, we decided to keep information about the vulnerability within our company — until now, when the issue has been fully resolved.
Dealing with security issues is a constant in any type of computing or cloud-hosting provider. At Slicehost, we work to provide you with the safest, most-stable environment possible. We regularly consult with independent security consultants. We employ a large and growing staff of security specialists and IT engineers. We are proud of their work in repairing this vulnerability, and grateful for your patience.
If you have questions, please reach out to your support team. We are here to serve you.
Slicehost Support Support@Slicehost.com
Summary:
The problem: When provisioning a disk, the provider reuses an old disk that has old customer data on it. Part of the disk is overwritten with the OS image but the rest is not.
The fix: Zero out the disk. Either when de-provisioning it, or when provisioning it.
The author's ability to turn 49 words into 2,543 words is quite impressive!
Also: we couldn't actually tell you this until now because we don't trust some of you.
One thing to consider here is keeping sensitive information on virtualized hosting setups block-level (full-disk) encrypted, for instance with Truecrypt.
Obviously it might seem silly to make this comment here, but would this really protect in many cases besides this one? It seems like almost every time the intrusion would occur somewhere where the intruder will have access to the mounted encrypted disk, no?
Full disk encryption is unlikely to protect you from anything other than custody issues at your hosting provider.
The idea behind this exploit:
The area on the SAN occupied by your disk contained a simple disk image. When your image is deleted and another is created overlapping the same spot on the host storage, they can look at their free space to see the contents of your disk.
If your disk happened to be encrypted, their free space would appear to contain garbage data.
Encrypting your disk would do you little good versus your provider if your VM can boot without your intervention... someone could just boot your VM to look inside.
Nothing you can reasonably do protects against a malicious (or compromised) provider; these are all controls that protect against provider mistakes.
A very interesting article about data leakage in cloud resource allocation. I'm impressed by the steps taken by all parties involved. Security vulnerabilities will continue to creep up over time as they do in most systems/services and the seriousness with which providers treat them is ever more important. What we have to remember is it's not just individuals hosting their own content that are at risk. More and more startups are utilizing cloud infrastructure which means we as consumers must be aware of where our personal data is going. I would love to see some sort of auditory process in place that verifies the security of cloud providers regularly and we as consumers having access to this information.
Yay LUKS/dm-crypt kernel level disc encryption.
We had need for short-term (several hours) EC2 compute nodes with sensitive data from customers. They'd boot up and encrypt all the drives using a random key without persisting it anywhere. Even so much as rebooting them made any data unusable. YMMV =) I'm thinking one could set the key using a start-up file so reboot would be possible.
This is not just limited to "cloud" services. Hosting companies recycle hard drives all the time. If you're lucky, they'll wipe it after they pull it out of your machine. If you're not, they won't.
It all comes down to what kind of recycling policies they have for old hardware. Ideally, all drives should be wiped and then stress-tested to avoid the problem where you eventually wind up with nothing but bad disks in the recycle loop. After all, good disks stay in production and thus exit the loop, whereas bad disks fail and come right back.
Why did this article so carefully not mention EC2 in the positive or negative?
This isn't "new" information, and many providers already wipe before reprovisioning space. Off the top of my head, I'm aware of three. The paper's sample size is frightfully small (VPS.NET, in particular, is not a shining star of cloud hosting). All you have to do to find out if this is the case is run this on new storage that you've requested from your provider:
This has nothing to do with "cloud", aside from rapidly reusing disks in that environment. This is the same problem with reusing disks in a datacenter, reusing disks at a dedicated host, reusing storage even inside a corporation, and so on. There is nothing "cloud-specific" about this in the slightest.Of course, they wait until near the bottom of the paper to tell you that, so the title of the paper seems like a needless cloud scare which is really a shared-environment problem. Clueful people in hosting figured this out a while ago (publicly-used hosting has been around a long, long time). Long enough ago that you can measure how clueful your provider is by the usefulness of this trick.
The cloud angle is important because, as a cloud user, I have no control over the situation. More so, most people just expect the provider to take care of them as a result.
Regardless, it's why I'm willing to pay a little more for a company that will be proactive.
At first I thought this was going to be the issue where people were sharing their EC2 instances and were leaving their keys or un-wiped copies of their private keys or tokens on images that others than stumble upon.