Yep.. I think I've verified it is currently impossible to access IMAP without asking for user credentials.

It would be nice to though. I can see all kinds of use cases for web apps integrating with gmail and other email providers in this way - without having to ask the user for their login/password.