I only use it as a cheap way to mentally background apps that won't be looked at as often (e.g. iTunes, running a server in Terminal).

I find that the focus model tends to break down in Spaces for any multi-window application like Finder, Chrome, Emacs and Terminal. If every window of an application is going to jump to the top then that just adds extra work to switching desktops.

That being said, I do like the Application->Window switching model on it's own and wouldn't trade this for Virtual Desktops.