They generally send you an advance email. I just had to migrate our Jenkins server a week or two ago because of this. I received something like 15 days notice on that one.
But obviously if there's a hard failure, they aren't always going to be able to give you the amount of time you'd want. Generally speaking, you should have accounted for this situation ahead of time in your engineering plans. Amazon EC2 doesn't have anything like vmotion, it's just a bunch of KVM virts.
If you're using the GUI, the first time you try a shutdown, it will do a normal request, but then if you go back and try it again while the first request is still pending, you should see the option for doing a hard restart. Try that and give it some time. Sometimes it takes an hour or two to get through. Otherwise, Amazon's tech support can help you.
Generally speaking, you should have accounted for this situation ahead of time in your engineering plans.
I believe this comes as a shock to most people the first time they receive this email, it was to us at least. When we signed up with amazon there was no guideline or advice saying "hey in a year or 2 your hardware might fail or need replaced, have a migration plan ready"
Perhaps it was our naivety, but we just thought hey, its the cloud, what could go wrong?! Now of course are are battle hardened
I had an instance fail (unresponsive then with that same stop/start delay) on the same day in two consecutive years. I remember because it happened to be Valentine's Day, and I had to break out the laptop for while to check on things. I always wondered if that was a completely random occurrence, or part of some maintenance schedule that randomly affected me twice.
The last place I worked has a policy of making zero changes in production Friday-Sunday or around holidays. It was one of their better practices.
I honestly thought most people knew that about EC2 as one of the core "trade offs" or engineering decision that allows the platform to be what it is, compared to a more traditional VPS provider.
That isn't a tradeoff of EC2: hardware fails... I've had servers from actual unmanaged server providers have random failures and need maintenance or need to be replaced as well. Hell: I've had my hosting provider tell me they are moving data centers before ;P. The real problem here is just that Amazon has made the idea of putting your servers in the cloud so easy that people who don't understand that the servers don't run on evil magic are able to use them.
EC2 used to terminate instances with no warning in many situations when it launched. It seems they've concluded most people didn't understand that, and avoid that whenever possible now.
But "cloud" compute services should in general be treated as less reliable per individual unit unless your provider explicitly explain to you why not (such as guaranteeing to use a high-availability distributed filesystem), as you no direct way of ascertaining status of the underlying hardware. You need to plan for failure regardless.
One of the reasons running jenkins on EC2 sucks for developers :( The data is stored on the machine, and there's a big risk of losing all your CI/CD infrastructure.
Have you given any thought to moving to something like https://circleci.com? [disclosure: I work there]
Comments
They generally send you an advance email. I just had to migrate our Jenkins server a week or two ago because of this. I received something like 15 days notice on that one.
But obviously if there's a hard failure, they aren't always going to be able to give you the amount of time you'd want. Generally speaking, you should have accounted for this situation ahead of time in your engineering plans. Amazon EC2 doesn't have anything like vmotion, it's just a bunch of KVM virts.
If you're using the GUI, the first time you try a shutdown, it will do a normal request, but then if you go back and try it again while the first request is still pending, you should see the option for doing a hard restart. Try that and give it some time. Sometimes it takes an hour or two to get through. Otherwise, Amazon's tech support can help you.
I believe this comes as a shock to most people the first time they receive this email, it was to us at least. When we signed up with amazon there was no guideline or advice saying "hey in a year or 2 your hardware might fail or need replaced, have a migration plan ready"
Perhaps it was our naivety, but we just thought hey, its the cloud, what could go wrong?! Now of course are are battle hardened
FWIW, they have a brief section about hardware failure in the Getting Started Guide [1]. I don't know how long it's been there.
[1] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-... (bottom of the page)
More accurately: "Your hardware my fail at any time."
I had an instance fail (unresponsive then with that same stop/start delay) on the same day in two consecutive years. I remember because it happened to be Valentine's Day, and I had to break out the laptop for while to check on things. I always wondered if that was a completely random occurrence, or part of some maintenance schedule that randomly affected me twice.
The last place I worked has a policy of making zero changes in production Friday-Sunday or around holidays. It was one of their better practices.
I honestly thought most people knew that about EC2 as one of the core "trade offs" or engineering decision that allows the platform to be what it is, compared to a more traditional VPS provider.
That isn't a tradeoff of EC2: hardware fails... I've had servers from actual unmanaged server providers have random failures and need maintenance or need to be replaced as well. Hell: I've had my hosting provider tell me they are moving data centers before ;P. The real problem here is just that Amazon has made the idea of putting your servers in the cloud so easy that people who don't understand that the servers don't run on evil magic are able to use them.
EC2 used to terminate instances with no warning in many situations when it launched. It seems they've concluded most people didn't understand that, and avoid that whenever possible now.
But "cloud" compute services should in general be treated as less reliable per individual unit unless your provider explicitly explain to you why not (such as guaranteeing to use a high-availability distributed filesystem), as you no direct way of ascertaining status of the underlying hardware. You need to plan for failure regardless.
One of the reasons running jenkins on EC2 sucks for developers :( The data is stored on the machine, and there's a big risk of losing all your CI/CD infrastructure.
Have you given any thought to moving to something like https://circleci.com? [disclosure: I work there]
That's why you use EBS and snapshots...
EBS is really slow for CI loads, in my experience. We deliberately don't touch it.
It wasn't that bad. I stopped and started an instance. Done.
EC2 uses Xen.