Is this fundamental, or just a problem with mapping these models to our current serially-bottlenecked compute architectures? Could a move to “hyperconverged infrastructure in-the-small” — striping DRAM or NVMe and tiny RISC cores together on a die, where each CPU gets its own storage (or, you might say, where each small cluster of storage cells has its own tiny CPU attached), such that one stick has millions of independent+concurrent [+slow+memory-constrained] processors — resolve these difficulties?
They require roughly the same amount of storage as modern ANN networks except that "neurons/synapses" may have some additional state that needs to be stored. Compared to the compute they require in relation to the compute needed for large-scale ANNs though, the storage is a lot.
Comments
Is this fundamental, or just a problem with mapping these models to our current serially-bottlenecked compute architectures? Could a move to “hyperconverged infrastructure in-the-small” — striping DRAM or NVMe and tiny RISC cores together on a die, where each CPU gets its own storage (or, you might say, where each small cluster of storage cells has its own tiny CPU attached), such that one stick has millions of independent+concurrent [+slow+memory-constrained] processors — resolve these difficulties?
They require roughly the same amount of storage as modern ANN networks except that "neurons/synapses" may have some additional state that needs to be stored. Compared to the compute they require in relation to the compute needed for large-scale ANNs though, the storage is a lot.