Run an IMAP server (on a separate host or VM) behind a locally hosted VPN - Wireguard is easy to set up and works fine for this - and you can have it both ways: mail accessible from all your clients no matter where you are. If your mail server is breached there won't be anything on it since everything is sucked up by the IMAP server. If your PC (where you currently store your mail) is compromised they'll be able to get access to everything so in that respect there is no change.
Comments
Run an IMAP server (on a separate host or VM) behind a locally hosted VPN - Wireguard is easy to set up and works fine for this - and you can have it both ways: mail accessible from all your clients no matter where you are. If your mail server is breached there won't be anything on it since everything is sucked up by the IMAP server. If your PC (where you currently store your mail) is compromised they'll be able to get access to everything so in that respect there is no change.