As I mentioned, before, I had imagined the network learning some fancy combination of logic gates to perform the whole addition process digitally, similarly to how a binary adder operates. This trick is yet another example of neural networks finding unexpected ways to solve problems.
My intuition is that this solution allows for some form of gradient approach to a solution, which is why it's unintuitive. We think about solutions as all or nothing and look for complete solutions.
Right, binary gates are discrete elements but neural networks operate on a continuous domain.
I'm reminded of the Feynman anecdote when he went to work for Thinking Machines and they gave him some task related to figuring out routing in the CPU network of the machine, which is a discrete problem. He came back with a solution that used partial differential equations, which surprised everyone.
Comments
My intuition is that this solution allows for some form of gradient approach to a solution, which is why it's unintuitive. We think about solutions as all or nothing and look for complete solutions.
Right, binary gates are discrete elements but neural networks operate on a continuous domain.
I'm reminded of the Feynman anecdote when he went to work for Thinking Machines and they gave him some task related to figuring out routing in the CPU network of the machine, which is a discrete problem. He came back with a solution that used partial differential equations, which surprised everyone.
The more interesting question is is it even possible to learn the logic gates solution through gradient descent?
You could riff off an approach similar to https://google-research.github.io/self-organising-systems/di...