The first question is why is there a divide between CPUs and GPUs in the first place. Yes, the gap is closing and both categories are adding features of one another but there still is a significant divide. IMO it has to do with Amdahl's law [0]. In that sense CPUs should be called Latency-Optimizing-Processors (LOPs) and GPUs should be called Throughput-Optimizing-Processors (TOPs).
More specifically [1] we could also call CPUs long / deep data dependency processors (LDDPs) and GPUs wide / flat data dependency processors (WDDPs).
Comments
The first question is why is there a divide between CPUs and GPUs in the first place. Yes, the gap is closing and both categories are adding features of one another but there still is a significant divide. IMO it has to do with Amdahl's law [0]. In that sense CPUs should be called Latency-Optimizing-Processors (LOPs) and GPUs should be called Throughput-Optimizing-Processors (TOPs).
More specifically [1] we could also call CPUs long / deep data dependency processors (LDDPs) and GPUs wide / flat data dependency processors (WDDPs).
[0]: https://en.wikipedia.org/wiki/Amdahl%27s_law [1]: https://en.wikipedia.org/wiki/Data_dependency
The observation that graphic hardware and general purpose CPUs converge and diverge is not new: http://cva.stanford.edu/classes/cs99s/papers/myer-sutherland... .
But as you observe, we are stuck in a local optimum where GPUs are optimized for throughput and CPUs for latency sensitive work.