For some GPUS, all code runs, no nested its, just flags to turn unused paths into no-ops. All I'm trying to say is even in your example, the optimal thing we could come up with depended on the problem at hand (graphics vs general purpose). Newer GPUs are different, but still, the ridicule of his question isn't appropriate, reasoning by analogy is error prone, and the real answer to his question is probably a lot more rigorous.
This is because GPUs are parallel and can't do nested ifs, not because representing code that way is efficient.
Even GPUs do sequential operations. You could never program the vast majority of algorithms like "if the input is exactly 100011110101... then output 100011001...".
If that seems silly, that is exactly how the proof that single layer NNs are perfectly general works. It proves that you can represent any series of if then statements like that in an NN. And people who don't understand the proof are mislead into thinking single layer NNs are just as good as deep NNs.
Comments
For some GPUS, all code runs, no nested its, just flags to turn unused paths into no-ops. All I'm trying to say is even in your example, the optimal thing we could come up with depended on the problem at hand (graphics vs general purpose). Newer GPUs are different, but still, the ridicule of his question isn't appropriate, reasoning by analogy is error prone, and the real answer to his question is probably a lot more rigorous.
This is because GPUs are parallel and can't do nested ifs, not because representing code that way is efficient.
Even GPUs do sequential operations. You could never program the vast majority of algorithms like "if the input is exactly 100011110101... then output 100011001...".
If that seems silly, that is exactly how the proof that single layer NNs are perfectly general works. It proves that you can represent any series of if then statements like that in an NN. And people who don't understand the proof are mislead into thinking single layer NNs are just as good as deep NNs.