Training tries to produce something that scores highly in training evaluations. With one data point, the evaluation is solely how closely the model output resembles the single input text.
Let's say you do that, and the training text is 58,100 tokens long. Let's say you ask the model to produce 58,101 tokens. Will it "reproduce [the] text verbatim"? No, it can't, because of the dissimilar requested length. Something "new" will come out.
It's also entirely possible that no matter how long you train, the model never converges on generating exactly the same output as its training data - you could end up with an average loss value of 0.001 instead of 0.0. It's not a perfectly deterministic process.
You're correct in principle, but in reality even with limited training data real-world models produce something that isn't exactly their training, especially when sampled stochastically. They're biased toward their training data, not forced to it.
Comments
If I train my 1000 billion parameter blank AI on this text and only this text over and over again, we may get 99% there.
You'll get 100% of the way there, because, lacking any other training, it will only ever be able to reproduce this text verbatim.
Hence, 100%, hence mission accomplished.
That's not how training works.
Training tries to produce something that scores highly in training evaluations. With one data point, the evaluation is solely how closely the model output resembles the single input text.
Let's say you do that, and the training text is 58,100 tokens long. Let's say you ask the model to produce 58,101 tokens. Will it "reproduce [the] text verbatim"? No, it can't, because of the dissimilar requested length. Something "new" will come out.
It's also entirely possible that no matter how long you train, the model never converges on generating exactly the same output as its training data - you could end up with an average loss value of 0.001 instead of 0.0. It's not a perfectly deterministic process.
You're correct in principle, but in reality even with limited training data real-world models produce something that isn't exactly their training, especially when sampled stochastically. They're biased toward their training data, not forced to it.
... It's never been tried. Look, I was actually joking when I initially replied, but now that I think about it, it's never been tried!
So now I am left wondering exactly what will happen if I train a new model from scratch, on only a single short corpus of 50k words.
Not fine-tuning an existing model, but literally creating a new model by feeding a blank (or random) matrix the same poem 100k times.