I don't see a situation where you are unable to get console access as an acceptable situation. You don't need to be using a smart card to require local console access if things go bad. What about if the SSH server fails or someone applies a bad firewall rule that prevents SSH access?