I've gone in and removed mine. I can't be certain but when I signed up (before it went fully public, I think) I don't remember it saying the e-mail would ever be visible (and I used the same unique e-mail address I use for all my Git work/commits). So it's well worth GitHub users double check what they have in their profile "just in case" they entered it at a time they though it'd be private.