Reading stuff like this (as a layman), diminishing these things as 'Next token predictors' seems absurdly reductive. At some point we'll need to concede that 'selection' is a better term for this than prediction.
which in no way challenge the established view that these bots are next-token predictors.
I mean, of course they are, that's literally what the inference loop does. You can look at the source of your favorite model runner and you'll see exactly that.
What I find misleading about this term is that it focuses attention on the "next token" part and glosses over the "prediction" part as some sort of unspecified "statistical algorithm" - even though this is where most of the work happens and where the interesting questions are.
There are other next token prediction algorithms such as markov chains or HMMs that also "fit the same interface", but are vastly simpler than what LLMs use.
I've seen various posters talk about "stochastic parrots" or about how LLMs were just "using very simple word statistics" to get their results, which sounded to me very much as if they thought of LLMs simply as glorified markov chains. I would consider that a big misunderstanding.
"In machine learning, the term stochastic parrot is a metaphor that frames large language models as systems that statistically mimic text without real understanding."
https://en.wikipedia.org/wiki/Stochastic_parrot
Comments
Reading stuff like this (as a layman), diminishing these things as 'Next token predictors' seems absurdly reductive. At some point we'll need to concede that 'selection' is a better term for this than prediction.
This shows a deep misunderstanding of the paper's claims, which in no way challenge the established view that these bots are next-token predictors.
Regardless, if all you want is a next-token selector, save your money and roll a die.
No, this shows an appreciation of the symbolic richness behind that token 'prediction' which the paper leads on.
Tell me, where is the emergent symbology guiding that dice?
The paper claims no symbolic richness beyond that evident from the undisputed next-token prediction.
There's none. That's my point.
I mean, of course they are, that's literally what the inference loop does. You can look at the source of your favorite model runner and you'll see exactly that.
What I find misleading about this term is that it focuses attention on the "next token" part and glosses over the "prediction" part as some sort of unspecified "statistical algorithm" - even though this is where most of the work happens and where the interesting questions are.
I've seen nothing to suggest it misleads anyone else.
There are other next token prediction algorithms such as markov chains or HMMs that also "fit the same interface", but are vastly simpler than what LLMs use.
I've seen various posters talk about "stochastic parrots" or about how LLMs were just "using very simple word statistics" to get their results, which sounded to me very much as if they thought of LLMs simply as glorified markov chains. I would consider that a big misunderstanding.
No misunderstanding at all.
"In machine learning, the term stochastic parrot is a metaphor that frames large language models as systems that statistically mimic text without real understanding." https://en.wikipedia.org/wiki/Stochastic_parrot
"Large Language Models as Markov Chains" https://arxiv.org/abs/2410.02724