I seriously think if you can't enjoy playing with the people you work with, something is wrong with your team.

Personally, I'm an intorvert and just working together with people makes me tired enough to not want to participate in extracurriculum activities with the team, even if it is entertainment. If it means something is wrong with my team, then something was wrong with my team at every place where I worked. I do some activities with the team just not to seem too weird - it's not a huge effort and not a big discomfort, but usually I would rather prefer to be elsewhere.