"A cursory look at the master key file date told me that no, it wouldn’t be OK at all. The file sitting on cryptoserv wasn’t from last night, it was created just a few minutes ago. During the Salt-n-Pepa-themed push from stage, we overwrote the master key file with the stage version... Not only that, the push script appears to have also wiped out the backup of the old key, so the database backups we have encrypted with the old key are likely useless."
So, despite all the planning and premeditation, they ended up doing exactly what AstralStorm says. An undesirable outcome does not stop being an accident if it is an unanticipated and unintended consequence of a planned action. Being in a crisis because you have forgotten how you planned to avoid it also counts as an accident.
As for the "Salt-n-Pepa-themed push": personally, I would have avoided all possible distractions at a time like this. Turning off phones (except maybe one, monitored by someone for whom that was their only job) would not have been unreasonable.
Comments
"There was nothing to do, except to hit rewind...
"A cursory look at the master key file date told me that no, it wouldn’t be OK at all. The file sitting on cryptoserv wasn’t from last night, it was created just a few minutes ago. During the Salt-n-Pepa-themed push from stage, we overwrote the master key file with the stage version... Not only that, the push script appears to have also wiped out the backup of the old key, so the database backups we have encrypted with the old key are likely useless."
So, despite all the planning and premeditation, they ended up doing exactly what AstralStorm says. An undesirable outcome does not stop being an accident if it is an unanticipated and unintended consequence of a planned action. Being in a crisis because you have forgotten how you planned to avoid it also counts as an accident.
As for the "Salt-n-Pepa-themed push": personally, I would have avoided all possible distractions at a time like this. Turning off phones (except maybe one, monitored by someone for whom that was their only job) would not have been unreasonable.