The problem with deep learning is that the functions are not fully specified. E.g. you can train the truth table of the XOR function. But you can't train on all possible cat pictures. The question is: will a single layer even train well on a subset of all possible cat pictures?
> But you can't train on all possible cat pictures.
I wanted to make a joke here that, sure you can, all you need is an infinite number of cat pictures and infinite time and compute, but actually you don't need that. Neural nets can do better at identifying cats in images than anything that has gone before and that has to be said.
The problem is that this better performance degrades precipitiously with any distance from the standard benchmark datasets that may contain cats. In the real world, with its infinite variation of not only cats but also cat poses, environments, lighting conditions, etc etc, it's a much worse deal and as I say in another comment, good luck knowing how well SOTA approaches do at it - we'll never know.
For instance, poverty of the stimulus (basically what you describe, but for speech) is not a problem when learning a language. Should it be an obstacle for pictures?
Comments
Yes. A sufficiently wide neural network with a single hidden layer can approximate any function arbitrarily closely.
The problem with deep learning is that the functions are not fully specified. E.g. you can train the truth table of the XOR function. But you can't train on all possible cat pictures. The question is: will a single layer even train well on a subset of all possible cat pictures?
I wanted to make a joke here that, sure you can, all you need is an infinite number of cat pictures and infinite time and compute, but actually you don't need that. Neural nets can do better at identifying cats in images than anything that has gone before and that has to be said.
The problem is that this better performance degrades precipitiously with any distance from the standard benchmark datasets that may contain cats. In the real world, with its infinite variation of not only cats but also cat poses, environments, lighting conditions, etc etc, it's a much worse deal and as I say in another comment, good luck knowing how well SOTA approaches do at it - we'll never know.
For instance, poverty of the stimulus (basically what you describe, but for speech) is not a problem when learning a language. Should it be an obstacle for pictures?
The missing bit of theory is in explaining the dynamics of training deeper networks
Exactly. Just because such a function exists does not necessarily mean that you can find it.