Great explanation, but we should distinguish between memory bandwidth -- how many bytes we can read or write per second -- and memory latency, how long it takes to load or store some memory location. Bandwidth is actually pretty fast these days; the latency is what sucks.
So, for example, if you want to add two large streams of numbers (e.g. dense matrices) together, a CPU can do this pretty quickly, because it can fetch the memory in bulk and not need to incur much latency penalty. (It can also avoid polluting the cache, with the correct hint instructions.)
On more typical workloads, though, where you've got a lot of harder-to-predict memory access, what would really come in handy is lower memory latency. And if this type of RAM works, it will give us both: huge bandwidth, and dramatically lower latency.
The system stack, instead of being HD->RAM->L3->L2->L1->registers, could look more like "Hard drive -> Giant shared non-uniform nonvolatile L3 cache -> shared or core-local L2 cache -> core-local L1 cache -> registers".
Comments
Great explanation, but we should distinguish between memory bandwidth -- how many bytes we can read or write per second -- and memory latency, how long it takes to load or store some memory location. Bandwidth is actually pretty fast these days; the latency is what sucks.
So, for example, if you want to add two large streams of numbers (e.g. dense matrices) together, a CPU can do this pretty quickly, because it can fetch the memory in bulk and not need to incur much latency penalty. (It can also avoid polluting the cache, with the correct hint instructions.)
On more typical workloads, though, where you've got a lot of harder-to-predict memory access, what would really come in handy is lower memory latency. And if this type of RAM works, it will give us both: huge bandwidth, and dramatically lower latency.
The system stack, instead of being HD->RAM->L3->L2->L1->registers, could look more like "Hard drive -> Giant shared non-uniform nonvolatile L3 cache -> shared or core-local L2 cache -> core-local L1 cache -> registers".