While, as other people have said, if you need to be convinced you shouldn't do it, I would say that if you're seriously considering doing so (which you probably are, given how much thought your second paragraph shows that you've put into it), then you've already decided to do so. Now you're just trying to rationalize it, to convince yourself that it's a good idea.

So just do it. The worst that can happen is failure, and it's far better to regret trying and failing than it is to regret not taking a chance.