To echo what others are saying, I don't think this will hurt your career, if anything it should help and should give you the opportunity to meet new people, start working on projects you are more passionate about, and gain new skills. All making you much more 'employable' I would say.

But more importantly I think I'd ask myself, if not now, when? I have a feeling its never going to get easier to go out on a limb, so the question is more will you be happy looking back if you didn't give it a shot? Live a little.

:)