Unfortunately those application-specific passwords aren't particularly application specific, at least in Google's implementation -- any of them can be used for anything.

If someone built a system which could restrict passwords or keys by some kind of capabilities (e.g. my Adium gtalk password could only be used to authenticate to Google's Jabber servers), that would be useful. It would be complex to manage, especially as your applications change over time, but not impossible.