By the time I finished my Ph.D. in 1997 I was stone cold certain that SVM's were equivalent to NN's, and that 3 layers could do anything. I saw Y.Lecuns work on handwriting and thought it an oddity. I gave up on machine learning as I thought it was all done and concentrated on applications.
In 2011 I was staggered to realise what a load of bollocks I had been bongoogled into believing in, and annoyed.
I am never going to fall for the "if we assume x, and y and only talk about this bit of the system then it's all the same" line again. It may be mathematically nice to prove things with simplifying assumptions, but the ICML community basically walked up a blind ally because of this desire for rigor and elegance in preference to reality.
ML is not mathematics, because it's tied to data, and data is a product of nature. The study of nature is Science. This is the difference between String Theory and Physics, and it's the difference between Computer Science and ML as well.
Isn't it still the case mathematically that 3 layers can do theoretically anything? Which of course may be a different claim that they can do that practically.
The problem with deep learning is that the functions are not fully specified. E.g. you can train the truth table of the XOR function. But you can't train on all possible cat pictures. The question is: will a single layer even train well on a subset of all possible cat pictures?
> But you can't train on all possible cat pictures.
I wanted to make a joke here that, sure you can, all you need is an infinite number of cat pictures and infinite time and compute, but actually you don't need that. Neural nets can do better at identifying cats in images than anything that has gone before and that has to be said.
The problem is that this better performance degrades precipitiously with any distance from the standard benchmark datasets that may contain cats. In the real world, with its infinite variation of not only cats but also cat poses, environments, lighting conditions, etc etc, it's a much worse deal and as I say in another comment, good luck knowing how well SOTA approaches do at it - we'll never know.
For instance, poverty of the stimulus (basically what you describe, but for speech) is not a problem when learning a language. Should it be an obstacle for pictures?
The point is that a three layer network can approximate any function, and back propagation can be used to train it to do so. You could train a 3 layer net to do imagenet, I think it was Minsky that showed the first bit of that in "Perceptrons", and Hinton showed it in his backprop paper with Rumelheart (?) but I can't be arsed to look it up, someone will pop up and show that it was actually Leonardo Da Vinci or that German fella who has apparently invented everything ever and never tires of telling everyone about it no matter how hard we all ignore him.
However, clever people (Hinton, Lecun, others v.smart and determined and all should get a lot of credit) invented some tricks to train networks faster (and carefully showed that they really worked, despite what people said about them, especially people at ICML), and computers got many faster, and it became obvious (which it was to anyone who both could do the sums and had the imagination to think about it - which I didn't, at all, sad for me) that broad networks are much much slower to train than deep networks for an equivalent number of neurons (if you use tricks like regularization and dropout) and this makes large deep networks feasible.
I did play with deep networks in like 1994, because I wrote a C program to let me write neural networks - and I could do anything I liked with it on my Sun workstation and my supervisors had no idea about anything. However, I set this "deep" network off training to see what would happen and went to the pub. When I came back I interrupted it and looked at the model file to figure out how much the parameters had changed since initialization (I didn't have any of the good small value initialization stuff, I think I set them to 0) and they had changed so little that I did some sums and decided that the model would not be finished training before I finished grad school if I finished at all.
So, I went off and built various types of booster on an MPI network that most of my colleagues didn't realize was running on their nice Sun workstations instead. Talking to other people in pubs at conferences has since convinced me that most of the grad students working on ML in the 90's did the same thing.
A while ago I put a stack of early 80's AI books (~1982) on my son's desk and said "Have a look through these and tell me what you think".
About thirty minutes later he came back and said something like: "Shit! That's all the stuff we are doing now!".
To which I said: "Yeah. Computers got faster and we have lots more storage and memory. Beyond that, I don't think we moved forward very much at all."
If you look though 30 to 40 year old literature on AI you will find pretty much everything you'd expect in a modern book on AI. Sure, yes, there have been advances, of course. I would challenge anyone to find 40 years of advances when comparing AI then and now.
The best hypothesis I can put on the table is that speed and massive storage made a range of applications possible over time. With those exhausted, we hit the wall because we are essentially working with 30 to 40 year old AI technology.
I feel similarly. Much worse, I cynically worry sometimes that the criticisms of the 60s/70s "AI Boom" are what got "lost" rather than any major advancements were "gained" since. A lot of 60s/70s optimism indeed got eaten by a general sense of "if we just had more storage and speed and larger datasets" (as we'd suspect from what we saw in recent improvements), but some of it got eaten by "it produces reasonable looking results but here's all the reasons it's a sandcastle of sparkling statistics that falls apart as the training set grows worse". I have a worrying gut feeling sometimes that so much of current ML enthusiasm missed some of those 70s/80s memos on reasons to be concerned about applying some of these "sandcastles" to the "hurricane" of modern data. We may have only massively scaled garbage in, garbage out more than anything else this AI/ML boom.
Interestingly I think that if you subtracted the hardware and went back to 1982 and put what has come since then on the table the interesting things would not be the smart tech of today (transformers, vector-dbs, MCMC,...) but would probably be answer sets. I think if they had come along in the early 80's then we would all be using logic programming and CS in general would look a lot different.
I don't think that there would be many games for mobile phones though.
Smart software engineers would not be dedicated to figuring out ways to get people to click on buttons or how to get people into virtual food fights for profit!
I can't even imagine what progress might look like if all those brains were put to better use.
Comments
By the time I finished my Ph.D. in 1997 I was stone cold certain that SVM's were equivalent to NN's, and that 3 layers could do anything. I saw Y.Lecuns work on handwriting and thought it an oddity. I gave up on machine learning as I thought it was all done and concentrated on applications.
In 2011 I was staggered to realise what a load of bollocks I had been bongoogled into believing in, and annoyed.
I am never going to fall for the "if we assume x, and y and only talk about this bit of the system then it's all the same" line again. It may be mathematically nice to prove things with simplifying assumptions, but the ICML community basically walked up a blind ally because of this desire for rigor and elegance in preference to reality.
ML is not mathematics, because it's tied to data, and data is a product of nature. The study of nature is Science. This is the difference between String Theory and Physics, and it's the difference between Computer Science and ML as well.
Isn't it still the case mathematically that 3 layers can do theoretically anything? Which of course may be a different claim that they can do that practically.
Yes. A sufficiently wide neural network with a single hidden layer can approximate any function arbitrarily closely.
The problem with deep learning is that the functions are not fully specified. E.g. you can train the truth table of the XOR function. But you can't train on all possible cat pictures. The question is: will a single layer even train well on a subset of all possible cat pictures?
I wanted to make a joke here that, sure you can, all you need is an infinite number of cat pictures and infinite time and compute, but actually you don't need that. Neural nets can do better at identifying cats in images than anything that has gone before and that has to be said.
The problem is that this better performance degrades precipitiously with any distance from the standard benchmark datasets that may contain cats. In the real world, with its infinite variation of not only cats but also cat poses, environments, lighting conditions, etc etc, it's a much worse deal and as I say in another comment, good luck knowing how well SOTA approaches do at it - we'll never know.
For instance, poverty of the stimulus (basically what you describe, but for speech) is not a problem when learning a language. Should it be an obstacle for pictures?
The missing bit of theory is in explaining the dynamics of training deeper networks
Exactly. Just because such a function exists does not necessarily mean that you can find it.
What do you (and parent) mean by "do"? Inference, or training, or both?
The point is that a three layer network can approximate any function, and back propagation can be used to train it to do so. You could train a 3 layer net to do imagenet, I think it was Minsky that showed the first bit of that in "Perceptrons", and Hinton showed it in his backprop paper with Rumelheart (?) but I can't be arsed to look it up, someone will pop up and show that it was actually Leonardo Da Vinci or that German fella who has apparently invented everything ever and never tires of telling everyone about it no matter how hard we all ignore him.
However, clever people (Hinton, Lecun, others v.smart and determined and all should get a lot of credit) invented some tricks to train networks faster (and carefully showed that they really worked, despite what people said about them, especially people at ICML), and computers got many faster, and it became obvious (which it was to anyone who both could do the sums and had the imagination to think about it - which I didn't, at all, sad for me) that broad networks are much much slower to train than deep networks for an equivalent number of neurons (if you use tricks like regularization and dropout) and this makes large deep networks feasible.
I did play with deep networks in like 1994, because I wrote a C program to let me write neural networks - and I could do anything I liked with it on my Sun workstation and my supervisors had no idea about anything. However, I set this "deep" network off training to see what would happen and went to the pub. When I came back I interrupted it and looked at the model file to figure out how much the parameters had changed since initialization (I didn't have any of the good small value initialization stuff, I think I set them to 0) and they had changed so little that I did some sums and decided that the model would not be finished training before I finished grad school if I finished at all.
So, I went off and built various types of booster on an MPI network that most of my colleagues didn't realize was running on their nice Sun workstations instead. Talking to other people in pubs at conferences has since convinced me that most of the grad students working on ML in the 90's did the same thing.
A while ago I put a stack of early 80's AI books (~1982) on my son's desk and said "Have a look through these and tell me what you think".
About thirty minutes later he came back and said something like: "Shit! That's all the stuff we are doing now!".
To which I said: "Yeah. Computers got faster and we have lots more storage and memory. Beyond that, I don't think we moved forward very much at all."
If you look though 30 to 40 year old literature on AI you will find pretty much everything you'd expect in a modern book on AI. Sure, yes, there have been advances, of course. I would challenge anyone to find 40 years of advances when comparing AI then and now.
The best hypothesis I can put on the table is that speed and massive storage made a range of applications possible over time. With those exhausted, we hit the wall because we are essentially working with 30 to 40 year old AI technology.
I feel similarly. Much worse, I cynically worry sometimes that the criticisms of the 60s/70s "AI Boom" are what got "lost" rather than any major advancements were "gained" since. A lot of 60s/70s optimism indeed got eaten by a general sense of "if we just had more storage and speed and larger datasets" (as we'd suspect from what we saw in recent improvements), but some of it got eaten by "it produces reasonable looking results but here's all the reasons it's a sandcastle of sparkling statistics that falls apart as the training set grows worse". I have a worrying gut feeling sometimes that so much of current ML enthusiasm missed some of those 70s/80s memos on reasons to be concerned about applying some of these "sandcastles" to the "hurricane" of modern data. We may have only massively scaled garbage in, garbage out more than anything else this AI/ML boom.
Interestingly I think that if you subtracted the hardware and went back to 1982 and put what has come since then on the table the interesting things would not be the smart tech of today (transformers, vector-dbs, MCMC,...) but would probably be answer sets. I think if they had come along in the early 80's then we would all be using logic programming and CS in general would look a lot different.
I don't think that there would be many games for mobile phones though.
Most of all:
Smart software engineers would not be dedicated to figuring out ways to get people to click on buttons or how to get people into virtual food fights for profit!
I can't even imagine what progress might look like if all those brains were put to better use.