That assumes that stealing your private key is the same thing as sending arbitrary packets from your machine, otherwise things like restricting subnets/ip addresses and port knocking could still be useful layers. How different exfil of a private key from being able to send arbitrary packets on the user's machine is...probably not that different under most circumstances I'd guess.