The big shortcoming of the article is that it doesn't answer this question. Why even bother?
Is there much mystery?
There is much mystery!
Why do some network architectures work better than others even with the same number of parameters?
Are all network architectures equivalent? In the sense of, say I have a choice of feed-forward convnet vs fully-connected vs MLP mixer vs transformer vs whatever else you want. Can I take my working transformer and get an equivalent convnet that trains the same way? Or is there something special about one or more of these architectures?
Are there non-network systems that have the same properties as deep networks? Maybe they're better?
What even is the space of non-equivalent models? Right now we basically just try stuff and hope to see some improvement. That's really unacceptable. If we had theory to tell us if two architectures are "the same" (there are many nuances here, just as there are when you are say comparing two Turing machines), we could talk about the space of all models.
When things work or don't work, is it because of some property of our optimization procedure or something inherent in the network initialization or architecture?
Why is optimizing some architectures so much easier than others? Transformers are great, but they're very unstable and tricky to train. As we add layers or modify a transformer, what do we do to update our training regime?
How much data do I need to train a network that does X? Can we predict this form some property of the task and the network? What do I know that I've saturated. If I've saturated what do I need to do to get improvements from more data?
Then we get to more serious problems. There are things that networks are terrible at, like long-range inferences and compositionality. Why is this? We have some fixes to some network architectures, but how do we generalize these?
And more broadly. We know basically nothing at all about recurrent networks. We thought they were useful, but then we essentially gave up on them completely with transformers. What's the deal? Are recurrent networks special? Are they approximated so well that they don't matter at all?
This all guides us toward a stranger mystery. We know the brain has recurrent connections. Actually, the visual system in your brain has more feedback connections than feedforward connections. Why don't we need to be able to build models that have these?
This just starts to scratch the surface about what's mysterious here. We're in dire need of some guiding theory, but it remains to be seen if this is it. So far theory has contributed nothing :(
Comments
The big shortcoming of the article is that it doesn't answer this question. Why even bother?
There is much mystery!
Why do some network architectures work better than others even with the same number of parameters?
Are all network architectures equivalent? In the sense of, say I have a choice of feed-forward convnet vs fully-connected vs MLP mixer vs transformer vs whatever else you want. Can I take my working transformer and get an equivalent convnet that trains the same way? Or is there something special about one or more of these architectures?
Are there non-network systems that have the same properties as deep networks? Maybe they're better?
What even is the space of non-equivalent models? Right now we basically just try stuff and hope to see some improvement. That's really unacceptable. If we had theory to tell us if two architectures are "the same" (there are many nuances here, just as there are when you are say comparing two Turing machines), we could talk about the space of all models.
When things work or don't work, is it because of some property of our optimization procedure or something inherent in the network initialization or architecture?
Why is optimizing some architectures so much easier than others? Transformers are great, but they're very unstable and tricky to train. As we add layers or modify a transformer, what do we do to update our training regime?
How much data do I need to train a network that does X? Can we predict this form some property of the task and the network? What do I know that I've saturated. If I've saturated what do I need to do to get improvements from more data?
Then we get to more serious problems. There are things that networks are terrible at, like long-range inferences and compositionality. Why is this? We have some fixes to some network architectures, but how do we generalize these?
And more broadly. We know basically nothing at all about recurrent networks. We thought they were useful, but then we essentially gave up on them completely with transformers. What's the deal? Are recurrent networks special? Are they approximated so well that they don't matter at all?
This all guides us toward a stranger mystery. We know the brain has recurrent connections. Actually, the visual system in your brain has more feedback connections than feedforward connections. Why don't we need to be able to build models that have these?
This just starts to scratch the surface about what's mysterious here. We're in dire need of some guiding theory, but it remains to be seen if this is it. So far theory has contributed nothing :(