If this can be used for training then this magic trick can be enough. Training a language model uses in principle also just one magic trick - trying to predict a next sentence in a large set of ordered sentences.