I'd recommend anyone interested in the NTK and its limitations to check out Greg Yang's work [1] and the recent book by Roberts, Yaida, and Hanin [2]. There's clearly a lot more work to be done on neural network scaling limits; the NTK is overly simplistic (though useful).
Comments
I'd recommend anyone interested in the NTK and its limitations to check out Greg Yang's work [1] and the recent book by Roberts, Yaida, and Hanin [2]. There's clearly a lot more work to be done on neural network scaling limits; the NTK is overly simplistic (though useful).
1: https://www.microsoft.com/en-us/research/people/gregyang/
2: https://arxiv.org/abs/2106.10165
Agreed, Greg Yang's work is really interesting.
Here's a reddit thread by Yang discussing his research into why NNs are not Kernel Machines [1].
[1] https://www.reddit.com/r/MachineLearning/comments/k8h01q/r_w...