Presumably you sign the emails rather than encrypt them?

That's correct, encrypted and signed. Replaying wouldn't be easy because the payload contains a timestamp. The main purpose was to limit the networks which can attempt to connect to ssh and still allow me to have a fallback if I'd happen to be outside of the "usual" network ranges.