It seems like it'd be possible to at least restrict your GitHub account from appearing linked to a commit unless it's a repo you've contributed to directly (by pushing or with a pull request).

That would need to be configurable, though -- it would kind of suck to lose credit for commits just because they weren't merged using GitHub's tools.