You're right, it's not. More attack surface area.

Sure, you could in theory have a highly distributed system with multiple keys, but then you can't use it day to day for monitoring communications, which is the whole purpose of the backdoor.

The government may be able to keep the nuclear codes safe in such a fashion, but it wouldn't if ten different government agencies wanted to use them on a daily basis.