Passphrases on keys really only provide some incidental security there, for keys that you don't happen to unlock while compromised. (Or the adversary might neglect to intercept your passphrases.)

Use a yubikey or smart card to level up from these.