I can't find any place explicitly saying this was done maliciously. A theory: this could also be a really bad accident where (for example) he works without a company provided computer and didn't clean out his old AWS profiles from previous company - ended up deleting resources from the wrong account.
Considering this, it also seems negligent to not provide computers to your employees. Employees are free from worry about leftovers of the employer's secret sauce and credentials on their machines and employers can eliminate a big source of trouble. It would pay off very quickly if even one big outage like this gets prevented.
While yes, this is a problem for several reasons (They should've taken care to clean any company IP off the laptop).
But the biggest, as others have already said is - why wern't his credentials revoked after leaving?
I can understand this at smaller companies, but Cisco has no excuse - they have enough people around that there's surely multiple people who's job it is to ensure that credentials are tied to a person, and that after a person leaves they're all revoked on anything approaching a production/customer-facing environment.
That seems like a lot of resources to delete... It’s “possible” but I’m not sure they I find it plausible. Is there some magic terraform or cloud formation that just destroys everything? And a Cisco engineer with devops like experience wouldn’t at least sniff around before running it? Doesn’t seem realistic.
Cisco should wear this too though, this is shockingly negligent. The only reason I can think of suggests a lot more problems and likely noncompliance with regulations and standards I’m sure they claim to comply with.
I was at a large company some time ago that had a script to delete any instances not tagged with a cost centre (from the testing account - they had planned to eventually run it in production but that was still a ways off).
One day an AWS API had an outage causing it to return an empty array for the tags list; the script deleted over 500 in-use instances.
I built an automated VPS system that would do things like create new VPS's, install various pieces of software, and delete test VPS's, etc.
I kept asking the sys admins to create a limited access account for my testing so that it flat couldn't delete existing customers VM's. I would walk into the office of the lead for that team once every few months and make the request again.
Until 1 day a bug in the VPS automation accidentally deleted a customer VPS thinking it was a failed deploy. They finally got around to giving me a limited account for dev/testing work.
It's scary how often this stuff falls through the cracks, even when employees KNOW it can happen.
classic. in the professional leagues whenever you do something like this it's properly vetted and even with that you want to bake a velocity check inside of it (ie don't erase more that 5 instances at a time). I cannot believe if you're expecting to see 3 instances go away but the tool says it's gonna do 500 you're going to be: sure thing boss. go ahead.
My money is that this guy was just negligent an did not remove his old credentials + used whatever thing he was using before without doublechecking. It's extremely plausible he had this setup for a test account and ran it with the wrong credentials.
On Cisco side this bad all around. Revoke the credentials. Audit the credentials periodically. Don't allow direct production access even for engineers that work there unless it's a live production issue or a deployment that's going on (and even then you allow them access to the tooling that does the deployment not to run whatever they want with : permissions)
A reused cloudformation template accidentally run on the wrong account could do an awful lot if it's set to tear down existing infra first so the env is clean
Sure. Essentially "for X in list-stacks; delete-stack $X". In this case it's only VMs (instances, I guess?) so it could be any cleanup script like "terminate not tagged", "terminate anything running over N days", or even just "delete-asg" which happened to have 456 instances in it.
Is there some magic terraform or cloud formation that just destroys everything?
Terraform has a nasty habit of instead of changing what you have into what you want, destroying what you have and then building what you want from scratch. It needs to be used very, very carefully.
Comments
I can't find any place explicitly saying this was done maliciously. A theory: this could also be a really bad accident where (for example) he works without a company provided computer and didn't clean out his old AWS profiles from previous company - ended up deleting resources from the wrong account.
Considering this, it also seems negligent to not provide computers to your employees. Employees are free from worry about leftovers of the employer's secret sauce and credentials on their machines and employers can eliminate a big source of trouble. It would pay off very quickly if even one big outage like this gets prevented.
While yes, this is a problem for several reasons (They should've taken care to clean any company IP off the laptop).
But the biggest, as others have already said is - why wern't his credentials revoked after leaving?
I can understand this at smaller companies, but Cisco has no excuse - they have enough people around that there's surely multiple people who's job it is to ensure that credentials are tied to a person, and that after a person leaves they're all revoked on anything approaching a production/customer-facing environment.
Especially 5 months after being fired. Steam should have had time to cool off.
i don't think he did this on purpose. it's probably the classic "ran with wrong credentials" story of one really unlucky negligent fellow.
That seems like a lot of resources to delete... It’s “possible” but I’m not sure they I find it plausible. Is there some magic terraform or cloud formation that just destroys everything? And a Cisco engineer with devops like experience wouldn’t at least sniff around before running it? Doesn’t seem realistic.
Cisco should wear this too though, this is shockingly negligent. The only reason I can think of suggests a lot more problems and likely noncompliance with regulations and standards I’m sure they claim to comply with.
I was at a large company some time ago that had a script to delete any instances not tagged with a cost centre (from the testing account - they had planned to eventually run it in production but that was still a ways off).
One day an AWS API had an outage causing it to return an empty array for the tags list; the script deleted over 500 in-use instances.
I built an automated VPS system that would do things like create new VPS's, install various pieces of software, and delete test VPS's, etc.
I kept asking the sys admins to create a limited access account for my testing so that it flat couldn't delete existing customers VM's. I would walk into the office of the lead for that team once every few months and make the request again.
Until 1 day a bug in the VPS automation accidentally deleted a customer VPS thinking it was a failed deploy. They finally got around to giving me a limited account for dev/testing work.
It's scary how often this stuff falls through the cracks, even when employees KNOW it can happen.
Tip: Set the termination protection flag on important EC2 instances to prevent accidental deletion.
classic. in the professional leagues whenever you do something like this it's properly vetted and even with that you want to bake a velocity check inside of it (ie don't erase more that 5 instances at a time). I cannot believe if you're expecting to see 3 instances go away but the tool says it's gonna do 500 you're going to be: sure thing boss. go ahead.
He "deployed code" that caused the deletions. I can definitely come up with many scenarios where something automatically removes VMs.
Edit: And https://news.ycombinator.com/item?id=24320495 provides a terraform command that is claimed to be potentially very destructive.
yes. there are tools out there that can wipe away everything: https://github.com/rebuy-de/aws-nuke
My money is that this guy was just negligent an did not remove his old credentials + used whatever thing he was using before without doublechecking. It's extremely plausible he had this setup for a test account and ran it with the wrong credentials.
On Cisco side this bad all around. Revoke the credentials. Audit the credentials periodically. Don't allow direct production access even for engineers that work there unless it's a live production issue or a deployment that's going on (and even then you allow them access to the tooling that does the deployment not to run whatever they want with : permissions)
A reused cloudformation template accidentally run on the wrong account could do an awful lot if it's set to tear down existing infra first so the env is clean
Sure. Essentially "for X in list-stacks; delete-stack $X". In this case it's only VMs (instances, I guess?) so it could be any cleanup script like "terminate not tagged", "terminate anything running over N days", or even just "delete-asg" which happened to have 456 instances in it.
Is there some magic terraform or cloud formation that just destroys everything?
Terraform has a nasty habit of instead of changing what you have into what you want, destroying what you have and then building what you want from scratch. It needs to be used very, very carefully.