Giving something a sinister name doesn’t change what it is.
Didnt Altman say a few years ago that the next frontier for training would be synthetic data? It’s what everyone turned to (Tiny Stories, etc.). Using existing LLMs to generate more training data for new ones is the obvious step.
LLMs are offered as document generators. This is what people do. They use them to generate documents. And then they train on those documents. The labs do it themselves. They didn’t ask when they trained on the internet. And now the Chinese - and researchers, and hobbyist, and businesses - dont ask when they train on LLM outputs. Especially when they paid for them.
What they call “distillation” is the very knowledge flywheel that we want in society. You buy a book, you may learn from it, and you may write a better one. The author got paid when yiu bought it. Society gets paid when the ideas spread and lead to the creation of more books, products and services, all of which increase the choices that everyone has.
Comments
Giving something a sinister name doesn’t change what it is.
Didnt Altman say a few years ago that the next frontier for training would be synthetic data? It’s what everyone turned to (Tiny Stories, etc.). Using existing LLMs to generate more training data for new ones is the obvious step.
LLMs are offered as document generators. This is what people do. They use them to generate documents. And then they train on those documents. The labs do it themselves. They didn’t ask when they trained on the internet. And now the Chinese - and researchers, and hobbyist, and businesses - dont ask when they train on LLM outputs. Especially when they paid for them.
What they call “distillation” is the very knowledge flywheel that we want in society. You buy a book, you may learn from it, and you may write a better one. The author got paid when yiu bought it. Society gets paid when the ideas spread and lead to the creation of more books, products and services, all of which increase the choices that everyone has.
Wait until they find out people use compiler output to write better compilers without asking for permission
It's compilers all the way down.
But so if there's distillation, and more distillation, and more distillation, what do we end up with ? A world of ELI5 ? "Grey zones are for losers" ?