Just firewall off most of the world when you won't be accessing your server from there.

Also look at fail2ban. Maybe portknocking also.