I've thought about this before as well. My "hobby" has been computer gaming as long as I can remember. I'm not doing a startup right now, but when I have, the time demands required me to make a choice. It sounds silly, but in reality we all find it difficult to walk away from something that's a major part of your identity.

The trick is to find a way to feed the part of your psyche that wants that activity / etc., maybe a substitute without the same drawbacks. Hiking instead of rock climbing, or chess instead of computer gaming. Virtual methadone...