Redundancy is a very good option, but the "right" level of modularization helps too. You probably don't want to make everything out of FPGAs, but if your individual modules can be flexibly rewired on the fly and they are granular enough in their functionality to build new things, that adds yet another level of redundancy, as you might be able to replace a broken thing with the right combination of working modules.