"A cursory look at the master key file date told me that no, it wouldn’t be OK at all. The file sitting on cryptoserv wasn’t from last night, it was created just a few minutes ago. During the Salt-n-Pepa-themed push from stage, we overwrote the master key file with the stage version... Not only that, the push script appears to have also wiped out the backup of the old key, so the database backups we have encrypted with the old key are likely useless."
So, despite all the planning and premeditation, they ended up doing exactly what AstralStorm says. An undesirable outcome does not stop being an accident if it is an unanticipated and unintended consequence of a planned action. Being in a crisis because you have forgotten how you planned to avoid it also counts as an accident.
As for the "Salt-n-Pepa-themed push": personally, I would have avoided all possible distractions at a time like this. Turning off phones (except maybe one, monitored by someone for whom that was their only job) would not have been unreasonable.
The oversight is that they developed and tested it on Linux, created the file on Linux as well, but used it on Solaris, using an operating system tool to enter the password that had different behaviour on Solaris (stripping anything beyond 8 characters in length because... nobody needs long passwords, right?).
It still happens today though, and subtle differences between computers or e.g. CPU architectures can and will still occur. It's really difficult or cost prohibitive to set up a DTAP or similar where every step is production-like; I can imagine it's similarly complex on modern day distributed systems / microservices.
So, premeditated or not, it counts as pushing development code into production. The accidentally part I believe is because someone didn't know any better.
Comments
Did you read the article? There was nothing accidental about this. It was done with a lot of planning and premeditation.
"There was nothing to do, except to hit rewind...
"A cursory look at the master key file date told me that no, it wouldn’t be OK at all. The file sitting on cryptoserv wasn’t from last night, it was created just a few minutes ago. During the Salt-n-Pepa-themed push from stage, we overwrote the master key file with the stage version... Not only that, the push script appears to have also wiped out the backup of the old key, so the database backups we have encrypted with the old key are likely useless."
So, despite all the planning and premeditation, they ended up doing exactly what AstralStorm says. An undesirable outcome does not stop being an accident if it is an unanticipated and unintended consequence of a planned action. Being in a crisis because you have forgotten how you planned to avoid it also counts as an accident.
As for the "Salt-n-Pepa-themed push": personally, I would have avoided all possible distractions at a time like this. Turning off phones (except maybe one, monitored by someone for whom that was their only job) would not have been unreasonable.
The oversight is that they developed and tested it on Linux, created the file on Linux as well, but used it on Solaris, using an operating system tool to enter the password that had different behaviour on Solaris (stripping anything beyond 8 characters in length because... nobody needs long passwords, right?).
It still happens today though, and subtle differences between computers or e.g. CPU architectures can and will still occur. It's really difficult or cost prohibitive to set up a DTAP or similar where every step is production-like; I can imagine it's similarly complex on modern day distributed systems / microservices.
There was no testing. No staging setup.
So, premeditated or not, it counts as pushing development code into production. The accidentally part I believe is because someone didn't know any better.
Not every company has a staging environment...
As the saying goes, everyone has a test environment, some are lucky enough to have a separate production environment.
Particularly since, by my recollection, 20 years ago a separate staging environment was the exception not the rule.