Then you've never actually Pair Programmed. It's not "someone looking over your shoulder", it's literally two people writing the code together, one person at the keyboard and the other person saying what's next.

That is literally what most people hate about it.