Wait, shouldn't you measure the information in output data (information about classes of samples) to evaluate overparameterization, not all the information in various features?
I.e. total overfitting can be achieved by ignoring most of the input values and using just as many pixels from the top left corner as you need to uniquely identify the source image (which is far less than the total number of pixels) and map it to an arbitrary class, and for a million images that should be theoretically possible with million-ish parameters; i.e. no matter how many pixels there were in each image, 138 million parameters for classification of a million images is overparameterized from the classic perspective of what overparameterization is (some generalization forced simply because it's theoretically impossible to fit each sample individually).
Comments
Wait, shouldn't you measure the information in output data (information about classes of samples) to evaluate overparameterization, not all the information in various features?
I.e. total overfitting can be achieved by ignoring most of the input values and using just as many pixels from the top left corner as you need to uniquely identify the source image (which is far less than the total number of pixels) and map it to an arbitrary class, and for a million images that should be theoretically possible with million-ish parameters; i.e. no matter how many pixels there were in each image, 138 million parameters for classification of a million images is overparameterized from the classic perspective of what overparameterization is (some generalization forced simply because it's theoretically impossible to fit each sample individually).