definitely always use a passphrase on the key. bonus, if you userify, than you only need to remember your userify console password to get into anywhere, since you can regen a key and replace on all your servers in seconds.