It's a ML term, inference basically means using the probability model you learned to draw "inferences" about a piece of data. In this context, it means giving the language model some context and using some method (either arg max sampling or something more sophisticated like beam search) to do what amounts to statistical auto implemention on it. As you might imagine, doing this with 530 GB of data at speed is quite energy intensive, even though there are things you can do to compress the model (distillation, pruning, compression/discretization) and specialised inference hardware.
Technically there is some very specific meaning to inference vs. prediction, but it's been heavily overloaded with meaning by now
Comments
When you say "inference", do you mean "interface", or is "inference" an ML term I'm not familiar with?
It's a ML term, inference basically means using the probability model you learned to draw "inferences" about a piece of data. In this context, it means giving the language model some context and using some method (either arg max sampling or something more sophisticated like beam search) to do what amounts to statistical auto implemention on it. As you might imagine, doing this with 530 GB of data at speed is quite energy intensive, even though there are things you can do to compress the model (distillation, pruning, compression/discretization) and specialised inference hardware.
Technically there is some very specific meaning to inference vs. prediction, but it's been heavily overloaded with meaning by now
First you train a model then you use it, "inference" is a fancy word for using the model.