This is interesting - I've never thought to investigate how ssh stores keys.

For managing passphrases, I strongly recommend the program keychain:

http://www.funtoo.org/Keychain

It's a thin wrapper around ssh-agent, but ultimately means I only need to enter my passphrase once.

The nifty feature is that you can add a command to your .login to clear your passphrase, so it's safe to use on remote hosts - if someone else is able to ssh in, they only get access to that host, not others that your ssh key allows access to.