Comment on Grokfast: Accelerated Grokking by Amplifying Slow GradientsparentComments−killerstorm2yIIRC it was observed in a training mode with weight decay. Perhaps a basin with proper generalization is more stable.
Comments
IIRC it was observed in a training mode with weight decay. Perhaps a basin with proper generalization is more stable.