They're not saying "size of the data set in bits", they're saying "number of items in the dataset". In speech and image recognition, it's normal to have more parameters than data points. This is a bit old, although it's still a very good architecture, but: GoogLeNet [0] has around 10M parameters, and was trained on 1.2M images.
In fact, the 2016 winner of a bunch of the ILSVRC challenges [1,2] was topologically basically the same as GoogLeNet.
EDIT: There's a perspective on machine learning which is basically just: "what if your model learns a hash-map". Check out Vapnik-Chervonenkis dimension.
Comments
They're not saying "size of the data set in bits", they're saying "number of items in the dataset". In speech and image recognition, it's normal to have more parameters than data points. This is a bit old, although it's still a very good architecture, but: GoogLeNet [0] has around 10M parameters, and was trained on 1.2M images.
In fact, the 2016 winner of a bunch of the ILSVRC challenges [1,2] was topologically basically the same as GoogLeNet.
EDIT: There's a perspective on machine learning which is basically just: "what if your model learns a hash-map". Check out Vapnik-Chervonenkis dimension.
[0] https://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf [1] https://arxiv.org/pdf/1601.05150v2.pdf [2] http://image-net.org/challenges/LSVRC/2016/results (CUImage)