Random sampling works well in base (true unsupervised) models, being only limited by their input distribution it's sampling from, I guess you can vaguely call that "sufficiently random" for certain uses, e.g. as a source of linguistic diversity. Any post-training with current methods will narrow the output distribution down, this is called mode collapse. It's not a fundamental limitation but it's hard to overcome and no AI shops care about it. Annoying LLM patterns in writing and media generation is a result of this.
Comments
I doubt LLMs can do random. Anyone have a good source on that?
Random sampling works well in base (true unsupervised) models, being only limited by their input distribution it's sampling from, I guess you can vaguely call that "sufficiently random" for certain uses, e.g. as a source of linguistic diversity. Any post-training with current methods will narrow the output distribution down, this is called mode collapse. It's not a fundamental limitation but it's hard to overcome and no AI shops care about it. Annoying LLM patterns in writing and media generation is a result of this.