Let me try to give a high-level explanation. As we have had success with using neural networks for different tasks, people began to ask what are the properties that make neural networks work so well.
When I mean work well, I mean a family of models which are generally easy to describe as well as be generally easy to train. Nearly all neural network models fit this bill as a they can be described with a loss function. A loss function being something which scores how accurately the current NN performs with a particular set of parameters. This loss function is then usually differentiable, and thanks to autodifferentiation tools, you can obtain the derivative of this function with respect to some parameters in this function. With this derivative, you can go ahead and run SGD to find the optimal parameters. We have now both empirical and theoretical evidence that SGD works fairly well for optimizing functions, particularly the high-dimensional ones neural networks describe.
So, the insight is we need a name to describe all these models which are differentiable and nice to train because of that. So now we can say what really ties CNNs, MLPs, RNNs, etc is not some biological metaphor, but that these are all expressible as some loss function we can find the minimum of using SGD.
Comments
Let me try to give a high-level explanation. As we have had success with using neural networks for different tasks, people began to ask what are the properties that make neural networks work so well.
When I mean work well, I mean a family of models which are generally easy to describe as well as be generally easy to train. Nearly all neural network models fit this bill as a they can be described with a loss function. A loss function being something which scores how accurately the current NN performs with a particular set of parameters. This loss function is then usually differentiable, and thanks to autodifferentiation tools, you can obtain the derivative of this function with respect to some parameters in this function. With this derivative, you can go ahead and run SGD to find the optimal parameters. We have now both empirical and theoretical evidence that SGD works fairly well for optimizing functions, particularly the high-dimensional ones neural networks describe.
So, the insight is we need a name to describe all these models which are differentiable and nice to train because of that. So now we can say what really ties CNNs, MLPs, RNNs, etc is not some biological metaphor, but that these are all expressible as some loss function we can find the minimum of using SGD.