Comment on Grokfast: Accelerated Grokking by Amplifying Slow GradientsComments−thesz2yGrokFast strongly reminds me of Stochastic Average Gradient Descent: https://www.cs.ubc.ca/~schmidtm/Courses/540-W19/L12.pdfBoth use averaging.
Comments
GrokFast strongly reminds me of Stochastic Average Gradient Descent: https://www.cs.ubc.ca/~schmidtm/Courses/540-W19/L12.pdf
Both use averaging.