I have so many questions about this now. For example, how often are people not distinguishing between amount of data versus number of cases?
To state your question more broadly. Data has structure that's why we can learn in the first place. Can we separately talk about the structure vs number of examples of the different structures in our data? In computer vision this would be sort of like talking about how much information is within an object class vs between object class.
We don't know. And we don't have the mathematical tools to talk about this today.
Do we really have any sense of how "effective number of parameters" scales with network features? This article had me wondering if peoples' understanding of these things is backward or something.
It's not backward, it's non-existent. We basically try things and when we have too many parameters we see that performance isn't increasing (or isn't increasing enough to be worth it, or with unsupervised methods and massive datasets we just max out the size of network we can train).
Comments
To state your question more broadly. Data has structure that's why we can learn in the first place. Can we separately talk about the structure vs number of examples of the different structures in our data? In computer vision this would be sort of like talking about how much information is within an object class vs between object class.
We don't know. And we don't have the mathematical tools to talk about this today.
It's not backward, it's non-existent. We basically try things and when we have too many parameters we see that performance isn't increasing (or isn't increasing enough to be worth it, or with unsupervised methods and massive datasets we just max out the size of network we can train).