I wonder whether you're dismissing today's GPUs too quickly. The way they work (and are programmed) today is about half way between CPUs and the linked architecture. They're generally applicable for an order of 10E3 parallel computations, with 10E5-10E6 threads they can be saturated. Whether you have computationally bounded or memory bandwidth bounded algorithms doesn't really matter (both is faster on GPU), what matters is a sufficiently long runtime for the parallelizable part of an application as well as not too much branching for computationally bounded kernels (there is a point where CPUs become faster when there's too many branches since for example on Kepler architecture for each branch, the neighbouring 192 cores are locked together).
The philosophy of today's GPU architecture is basically quite simple: Maximize memory throughput by using the fastest RAM that's still cheap enough for consumers, then maximize die space for the ALUs by letting bundles of them share scheduler, register blocks and cache. I was first very skeptical about this too, but to my experience it has proven quite effective - even parallel algorithms that are not ideal for this architecture still profit from the raw power, and they continue getting benefits when you buy new cards, in a fashion that's much closer to Moore's law than CPUs develop.
The architecture certainly isn't ideal and would be solved by an architecture like in your link (to which Parallela also comes quite close btw), and I can well imagine that this is where we're heading given another 5-10 years (see Parallela, to some extent Knight's Landing). However it's also feasible that the GPU's ALU maximisation game will win out, especially once 3D-stacked memory comes into play.
Since 2008 there have been many papers about NNs implemented on GPU and I'd love to know what's the current status there, especially compared to the very powerful Power8 architecture.
Comments
I wonder whether you're dismissing today's GPUs too quickly. The way they work (and are programmed) today is about half way between CPUs and the linked architecture. They're generally applicable for an order of 10E3 parallel computations, with 10E5-10E6 threads they can be saturated. Whether you have computationally bounded or memory bandwidth bounded algorithms doesn't really matter (both is faster on GPU), what matters is a sufficiently long runtime for the parallelizable part of an application as well as not too much branching for computationally bounded kernels (there is a point where CPUs become faster when there's too many branches since for example on Kepler architecture for each branch, the neighbouring 192 cores are locked together).
The philosophy of today's GPU architecture is basically quite simple: Maximize memory throughput by using the fastest RAM that's still cheap enough for consumers, then maximize die space for the ALUs by letting bundles of them share scheduler, register blocks and cache. I was first very skeptical about this too, but to my experience it has proven quite effective - even parallel algorithms that are not ideal for this architecture still profit from the raw power, and they continue getting benefits when you buy new cards, in a fashion that's much closer to Moore's law than CPUs develop.
The architecture certainly isn't ideal and would be solved by an architecture like in your link (to which Parallela also comes quite close btw), and I can well imagine that this is where we're heading given another 5-10 years (see Parallela, to some extent Knight's Landing). However it's also feasible that the GPU's ALU maximisation game will win out, especially once 3D-stacked memory comes into play.
Since 2008 there have been many papers about NNs implemented on GPU and I'd love to know what's the current status there, especially compared to the very powerful Power8 architecture.