Interesting theory, small sample. For me, it's more looking into the face of another is distracting. I involuntarily imagine time-lapse(forward & back) of a speaker/listener's face, note a facial scar/blemish and wonder briefly upon it's genesis or focus on their lips as they speak(I have some tone deafness that loses convo amidst background noise). I look for similarities and differences to myself and empathize with time's past and future effects. It is a huge distraction. Even more so when it's a stranger or adversarial encounter. As for staring into eyes, I find it unsettling when someone's focus jumps from my one eye to another, while I umdwrstand my static 'face stare' gives an impression I may be something of a dullard... eyes are only one indicator of body language, hyper-focus on them never gives me the complete, albeit distracting, picture.

edit: reword 2nd to last sentence.