Comment on OpenAI releasing new open model in coming months, seeks community feedbackparentComments−fooker1yIt's a kind of an open secret that there's no 'training' protocol for these state of the art models.Researchers behave like alchemists when training these models, and the actions are not really reproducible.−rockinghigh1yThey could provide access to the training code. It's useful for training smaller models or distilling larger ones. They don't need to release every details involved in tuning the optimization parameters during the pre-training stage.−fooker1yThere is no training 'code' that will get you anything close to a usable result.
Comments
It's a kind of an open secret that there's no 'training' protocol for these state of the art models.
Researchers behave like alchemists when training these models, and the actions are not really reproducible.
They could provide access to the training code. It's useful for training smaller models or distilling larger ones. They don't need to release every details involved in tuning the optimization parameters during the pre-training stage.
There is no training 'code' that will get you anything close to a usable result.