I'll hazard a guess that the root cause is nothing to do with the architecture of your system but everything to do with the exponential increase in SNAFUs cause by the extra complexity.

Almost :) It has more to do with the fact that testing such a setup under realistic conditions modelling all the potential failure modes is no substitute for the variety of ways in which a distributed system can fail. Network cards that still send but don't receive? Check (heartbeat thinks you're doing a-ok). Link between to DCs down, DCs themselves still up and running? Check... and so on.

Doing this right is extremely hard, and even the best of the best still get caught out (witness Amazon and Google outages, and I refuse to believe they don't know their stuff).

Hardware failures are rare, people failures are common, distributed systems are hard.