No Train No Gain:Revisiting Efficient Training Algrthm for Transformer-BasedLMarxiv.org 11 pointsfroster3 years ago1 commentSaveHideCopy link On HNComments−frosterOP3yRecent paper highlights the difficulty of creating a new optimizer as drop-in replacement. Sophia and Lion were recently proposed as superior alternatives to Adam, but appeared worse in an independent eval
Comments
Recent paper highlights the difficulty of creating a new optimizer as drop-in replacement. Sophia and Lion were recently proposed as superior alternatives to Adam, but appeared worse in an independent eval