Comment on Every Model Learned by Gradient Descent Is Approximately a Kernel MachineparentComments−talolard5yHmmm, I think that's not precise and my use of "architecture" was misleading.If we're thinking in terms of "universal aproximators", an RNN is a way to make a sequence of approximate functions for a sequence of inputs.But it's still a sequence of functions, not a single function.For a 1 layer network to have the same ability as an RNN (take an unbounded amount of context) it would need to have infinite width which is a no-go.
Comments
Hmmm, I think that's not precise and my use of "architecture" was misleading.
If we're thinking in terms of "universal aproximators", an RNN is a way to make a sequence of approximate functions for a sequence of inputs.
But it's still a sequence of functions, not a single function.
For a 1 layer network to have the same ability as an RNN (take an unbounded amount of context) it would need to have infinite width which is a no-go.