Comment on Large Concept Models: Language modeling in a sentence representation spaceparentComments−blackeyeblitzar1yIsn’t “sentence prediction” roughly the same as multi token prediction of sufficient length? In the end are we just talking about a change to hyper parameters or maybe a new hyper parameter that controls the granularity of “prediction length”?−mdp20211ymulti token prediction of sufficient lengthIs multi token prediction the same as predicting the embedding of a complex token (the articulation of those input tokens in a sentence)?−blackeyeblitzar1yTo be honest I don’t know. Maybe the only way to know is to build and measure all these variations.
Comments
Isn’t “sentence prediction” roughly the same as multi token prediction of sufficient length? In the end are we just talking about a change to hyper parameters or maybe a new hyper parameter that controls the granularity of “prediction length”?
Is multi token prediction the same as predicting the embedding of a complex token (the articulation of those input tokens in a sentence)?
To be honest I don’t know. Maybe the only way to know is to build and measure all these variations.