The point is that a three layer network can approximate any function, and back propagation can be used to train it to do so. You could train a 3 layer net to do imagenet, I think it was Minsky that showed the first bit of that in "Perceptrons", and Hinton showed it in his backprop paper with Rumelheart (?) but I can't be arsed to look it up, someone will pop up and show that it was actually Leonardo Da Vinci or that German fella who has apparently invented everything ever and never tires of telling everyone about it no matter how hard we all ignore him.
However, clever people (Hinton, Lecun, others v.smart and determined and all should get a lot of credit) invented some tricks to train networks faster (and carefully showed that they really worked, despite what people said about them, especially people at ICML), and computers got many faster, and it became obvious (which it was to anyone who both could do the sums and had the imagination to think about it - which I didn't, at all, sad for me) that broad networks are much much slower to train than deep networks for an equivalent number of neurons (if you use tricks like regularization and dropout) and this makes large deep networks feasible.
I did play with deep networks in like 1994, because I wrote a C program to let me write neural networks - and I could do anything I liked with it on my Sun workstation and my supervisors had no idea about anything. However, I set this "deep" network off training to see what would happen and went to the pub. When I came back I interrupted it and looked at the model file to figure out how much the parameters had changed since initialization (I didn't have any of the good small value initialization stuff, I think I set them to 0) and they had changed so little that I did some sums and decided that the model would not be finished training before I finished grad school if I finished at all.
So, I went off and built various types of booster on an MPI network that most of my colleagues didn't realize was running on their nice Sun workstations instead. Talking to other people in pubs at conferences has since convinced me that most of the grad students working on ML in the 90's did the same thing.
A while ago I put a stack of early 80's AI books (~1982) on my son's desk and said "Have a look through these and tell me what you think".
About thirty minutes later he came back and said something like: "Shit! That's all the stuff we are doing now!".
To which I said: "Yeah. Computers got faster and we have lots more storage and memory. Beyond that, I don't think we moved forward very much at all."
If you look though 30 to 40 year old literature on AI you will find pretty much everything you'd expect in a modern book on AI. Sure, yes, there have been advances, of course. I would challenge anyone to find 40 years of advances when comparing AI then and now.
The best hypothesis I can put on the table is that speed and massive storage made a range of applications possible over time. With those exhausted, we hit the wall because we are essentially working with 30 to 40 year old AI technology.
I feel similarly. Much worse, I cynically worry sometimes that the criticisms of the 60s/70s "AI Boom" are what got "lost" rather than any major advancements were "gained" since. A lot of 60s/70s optimism indeed got eaten by a general sense of "if we just had more storage and speed and larger datasets" (as we'd suspect from what we saw in recent improvements), but some of it got eaten by "it produces reasonable looking results but here's all the reasons it's a sandcastle of sparkling statistics that falls apart as the training set grows worse". I have a worrying gut feeling sometimes that so much of current ML enthusiasm missed some of those 70s/80s memos on reasons to be concerned about applying some of these "sandcastles" to the "hurricane" of modern data. We may have only massively scaled garbage in, garbage out more than anything else this AI/ML boom.
Interestingly I think that if you subtracted the hardware and went back to 1982 and put what has come since then on the table the interesting things would not be the smart tech of today (transformers, vector-dbs, MCMC,...) but would probably be answer sets. I think if they had come along in the early 80's then we would all be using logic programming and CS in general would look a lot different.
I don't think that there would be many games for mobile phones though.
Smart software engineers would not be dedicated to figuring out ways to get people to click on buttons or how to get people into virtual food fights for profit!
I can't even imagine what progress might look like if all those brains were put to better use.
Comments
What do you (and parent) mean by "do"? Inference, or training, or both?
The point is that a three layer network can approximate any function, and back propagation can be used to train it to do so. You could train a 3 layer net to do imagenet, I think it was Minsky that showed the first bit of that in "Perceptrons", and Hinton showed it in his backprop paper with Rumelheart (?) but I can't be arsed to look it up, someone will pop up and show that it was actually Leonardo Da Vinci or that German fella who has apparently invented everything ever and never tires of telling everyone about it no matter how hard we all ignore him.
However, clever people (Hinton, Lecun, others v.smart and determined and all should get a lot of credit) invented some tricks to train networks faster (and carefully showed that they really worked, despite what people said about them, especially people at ICML), and computers got many faster, and it became obvious (which it was to anyone who both could do the sums and had the imagination to think about it - which I didn't, at all, sad for me) that broad networks are much much slower to train than deep networks for an equivalent number of neurons (if you use tricks like regularization and dropout) and this makes large deep networks feasible.
I did play with deep networks in like 1994, because I wrote a C program to let me write neural networks - and I could do anything I liked with it on my Sun workstation and my supervisors had no idea about anything. However, I set this "deep" network off training to see what would happen and went to the pub. When I came back I interrupted it and looked at the model file to figure out how much the parameters had changed since initialization (I didn't have any of the good small value initialization stuff, I think I set them to 0) and they had changed so little that I did some sums and decided that the model would not be finished training before I finished grad school if I finished at all.
So, I went off and built various types of booster on an MPI network that most of my colleagues didn't realize was running on their nice Sun workstations instead. Talking to other people in pubs at conferences has since convinced me that most of the grad students working on ML in the 90's did the same thing.
A while ago I put a stack of early 80's AI books (~1982) on my son's desk and said "Have a look through these and tell me what you think".
About thirty minutes later he came back and said something like: "Shit! That's all the stuff we are doing now!".
To which I said: "Yeah. Computers got faster and we have lots more storage and memory. Beyond that, I don't think we moved forward very much at all."
If you look though 30 to 40 year old literature on AI you will find pretty much everything you'd expect in a modern book on AI. Sure, yes, there have been advances, of course. I would challenge anyone to find 40 years of advances when comparing AI then and now.
The best hypothesis I can put on the table is that speed and massive storage made a range of applications possible over time. With those exhausted, we hit the wall because we are essentially working with 30 to 40 year old AI technology.
I feel similarly. Much worse, I cynically worry sometimes that the criticisms of the 60s/70s "AI Boom" are what got "lost" rather than any major advancements were "gained" since. A lot of 60s/70s optimism indeed got eaten by a general sense of "if we just had more storage and speed and larger datasets" (as we'd suspect from what we saw in recent improvements), but some of it got eaten by "it produces reasonable looking results but here's all the reasons it's a sandcastle of sparkling statistics that falls apart as the training set grows worse". I have a worrying gut feeling sometimes that so much of current ML enthusiasm missed some of those 70s/80s memos on reasons to be concerned about applying some of these "sandcastles" to the "hurricane" of modern data. We may have only massively scaled garbage in, garbage out more than anything else this AI/ML boom.
Interestingly I think that if you subtracted the hardware and went back to 1982 and put what has come since then on the table the interesting things would not be the smart tech of today (transformers, vector-dbs, MCMC,...) but would probably be answer sets. I think if they had come along in the early 80's then we would all be using logic programming and CS in general would look a lot different.
I don't think that there would be many games for mobile phones though.
Most of all:
Smart software engineers would not be dedicated to figuring out ways to get people to click on buttons or how to get people into virtual food fights for profit!
I can't even imagine what progress might look like if all those brains were put to better use.