The problem with deep learning is that the functions are not fully specified. E.g. you can train the truth table of the XOR function. But you can't train on all possible cat pictures. The question is: will a single layer even train well on a subset of all possible cat pictures?
> But you can't train on all possible cat pictures.
I wanted to make a joke here that, sure you can, all you need is an infinite number of cat pictures and infinite time and compute, but actually you don't need that. Neural nets can do better at identifying cats in images than anything that has gone before and that has to be said.
The problem is that this better performance degrades precipitiously with any distance from the standard benchmark datasets that may contain cats. In the real world, with its infinite variation of not only cats but also cat poses, environments, lighting conditions, etc etc, it's a much worse deal and as I say in another comment, good luck knowing how well SOTA approaches do at it - we'll never know.
For instance, poverty of the stimulus (basically what you describe, but for speech) is not a problem when learning a language. Should it be an obstacle for pictures?
Comments
The problem with deep learning is that the functions are not fully specified. E.g. you can train the truth table of the XOR function. But you can't train on all possible cat pictures. The question is: will a single layer even train well on a subset of all possible cat pictures?
I wanted to make a joke here that, sure you can, all you need is an infinite number of cat pictures and infinite time and compute, but actually you don't need that. Neural nets can do better at identifying cats in images than anything that has gone before and that has to be said.
The problem is that this better performance degrades precipitiously with any distance from the standard benchmark datasets that may contain cats. In the real world, with its infinite variation of not only cats but also cat poses, environments, lighting conditions, etc etc, it's a much worse deal and as I say in another comment, good luck knowing how well SOTA approaches do at it - we'll never know.
For instance, poverty of the stimulus (basically what you describe, but for speech) is not a problem when learning a language. Should it be an obstacle for pictures?