Another interesting thing is that for training neural networks for image recognition purpose information about location of individual pixels is not necessary at all.

Convolutions use the spatial information. I'm less sure whether attention-based approaches typically use it.