Even so, I don't think there is any evidence that LLM performance degrades when it is trained on its own output, and there is no intuitive reason it should.
Intuitively, training a simple enough linear statistical model with its own output should be a NOP. But LLMs are anything but simple models, so I think the non-linearities may be synthesizing new useful information. Similarly to how all of maths can be synthesized from a few basic axioms with enough intelligence or computation.
Comments
But even so, the human picks the prompts and only publishes the AI outputs they think read nicely. There is information gain even in that.
At the moment that's probably true, but is it guaranteed to remain so?
Even so, I don't think there is any evidence that LLM performance degrades when it is trained on its own output, and there is no intuitive reason it should.
Why not? Training a model destroys information.
I have seen no evidence for that, only the opposite: https://arxiv.org/abs/2210.11610
Intuitively, training a simple enough linear statistical model with its own output should be a NOP. But LLMs are anything but simple models, so I think the non-linearities may be synthesizing new useful information. Similarly to how all of maths can be synthesized from a few basic axioms with enough intelligence or computation.