While I sort of agree that machine learning will end up as an experimental science, it's way, way too early to say whether the theory relating deep learning to kernel methods (e.g. Neural Tangent Kernels) will be useful or not.
As an example, just last week a (huge) paper [1] was put on arXiv that used these theoretical methods to analyze a bunch of common architecture building blocks (skip connections, normalization, etc), and then applied their theoretical findings to figure out how to train Resnet like models in similar training time without these seemingly "required" building blocks.
Deep Learning is still in its infancy in many ways, and this type of research takes time, slowly building on successive results.
Comments
While I sort of agree that machine learning will end up as an experimental science, it's way, way too early to say whether the theory relating deep learning to kernel methods (e.g. Neural Tangent Kernels) will be useful or not.
As an example, just last week a (huge) paper [1] was put on arXiv that used these theoretical methods to analyze a bunch of common architecture building blocks (skip connections, normalization, etc), and then applied their theoretical findings to figure out how to train Resnet like models in similar training time without these seemingly "required" building blocks.
Deep Learning is still in its infancy in many ways, and this type of research takes time, slowly building on successive results.
[1] https://arxiv.org/abs/2110.01765
When you wrote 'huge', I thought you meant huge potential impact; I wasn't expecting 172 pages.
team behind the paper is Deepmind/Google. It is probably worth a read.