I have done something similar -- I've run ssh-agent on my phone, then I ssh from my phone to my PC with the agent-forward (-A) option so that any ssh connections I do from my PC forwards the authentication requests to my phone. (This was with my Nokia N900 phone -- I'm working on setting up a similar thing for Android).