The (proprietary) driver issue is a big pain for people who want to use Linux for desktop as a workstation e.g. development. I would say that it is the biggest obstacle for adaption of Linux desktop.

Simply don't buy Nvidia. Apart from that there shouldn't be a lot of hardware requiring blobs on the desktop.