You can put the private key on a few HSMs, and have the HSM enforce a security policy (eg - access must be authorized by a quorum of operators). Never allow the private key to leave the HSM, ever again, except when initializing a new HSM (in which case, it had better be encrypted by the new HSM's unique key before leaving the old one).

Then you go fix your legacy firmware.