This is because GPUs are parallel and can't do nested ifs, not because representing code that way is efficient.
Even GPUs do sequential operations. You could never program the vast majority of algorithms like "if the input is exactly 100011110101... then output 100011001...".
If that seems silly, that is exactly how the proof that single layer NNs are perfectly general works. It proves that you can represent any series of if then statements like that in an NN. And people who don't understand the proof are mislead into thinking single layer NNs are just as good as deep NNs.
Comments
This is because GPUs are parallel and can't do nested ifs, not because representing code that way is efficient.
Even GPUs do sequential operations. You could never program the vast majority of algorithms like "if the input is exactly 100011110101... then output 100011001...".
If that seems silly, that is exactly how the proof that single layer NNs are perfectly general works. It proves that you can represent any series of if then statements like that in an NN. And people who don't understand the proof are mislead into thinking single layer NNs are just as good as deep NNs.