I think you're misunderstanding the point of the question. It's not about forcing anyone to work on a Sunday, it's about whether or not you feel like this person is someone you'd like to work with.

Lets say for whatever reason, this person is in the office on a Sunday. Does the idea of going and working with that person, all by yourself, sound terrible? If you can't be with that person all alone, there's no reason that they should be part of your team.