LLMs don’t ’replay variations on prior conversations’ though.
Predicting the probable token in a conversation requires predicting the probable subject of the conversation, predicting the interlocutors’ relationship and manner of speaking to one another, predicting the state of recollection, preference and taste of the speaker, predicting the speaker’s mental model…
If the LLM isn’t predicting all of those things then it will produce poor predictions of the next word; doing it well - and humans tend to agree that in a vast array of cases state of the art LLMs do predict tokens very well - requires that prediction model to predict all that context as well.
Alas, no it doesn't. Language induces this sort of anthropomorphism in people, I guess, so consider images.
Suppose I take a billion images of all the coffee cups in the world, at a set of angles on the cup, and then build an associative (ie., frequency) statistical model of their pixels (ie., statistical AI). Consider generating one pixel at a time, in sequence, through the image. My associative model tells me P(col of next pixel | all previous).
Now, I can generate coffee cups images similar to any variation or combination of the images in the dataset. Now, you might say, "well you can only do that if you have a model of a coffee cup" (rather than of pixels) -- if so, just generate a coffee cup at one of the angles not in the dataset. This will not happen, because the model has not been provided with enough information to do so.
Namely, the model does not know the distance from the camera, the camera lens parameters, the angle to the coffee cup, etc. So there's literally a very very large inifinity of possible objects at unseen angles. Consider that underneath a coffee cup, the bottom might be missing entirely, etc.
Now it will appear to know all of these things, because its just generating images with these same parameters (camera, angle, distance, etc.). But as soon as you want "a coffee further away than has been seen before", or "a coffee using a macro lens", etc. the whole thing will fall over.
It is you, the view, who attributes 3D knowledge to the model because under ordinary circumstances the cause of a photo is features of a 3D environment.
You’re saying this with confidence as if there isn’t a large body of working image and video generation algorithms out there that can produce physically plausible images of objects transposed into circumstances that don’t exist in their training set. A coffee using a macro lens for example.
Is it so hard to believe that such models have developed a sense for how light propagates through a scene, a sense for how physical objects change when viewed from different angles, a sense for how lens distortion interacts with light? For goodness’ sake, these same models have a sense of what Greg Rutkowski’s art style is - we are well beyond ‘they’re just remembering pixels from past coffeecups’
it so hard to believe that such models have developed a sense for how light propagates
Well, its not a matter or belief or otherwise. I'm a trained practitioner in statistics, AI, physics, and other areas and you can show trivially that you cannot learn light physics from pixel distributions.
Pixel distributions aren't stationary, and are caused by a very very large number of factors; likewise the physics of light for any given situation is subject to a large number of causes, all of them entirely absent from from the pixel distributions. This is a pretty trivial thing to show.
have a sense of what Greg Rutkowski’s art style is
Well what these models show is that when you have PBs of image data and TBs of associated text data, you can relate words and images together usefully. In particular, you can use patterns of text tokens to sample from image distributions, and combine and vary these samples to produce novel images.
The patterns in text and images are caused by people speaking, taking photos, etc. Those patterns necessarily obtain in any generated output. As in, if you train an LLM/etc. on how to speak, using vast amounts of conversational data, it cannot do anything other than appear to speak: that is the only thing the data distribution makes possible.
Likewise here, the image generator has a compressed representation of PBs of pixel data which can be sampled from using text. So when you say, "Greg Rutkowski" you select for a highly structured image space, whose structure the original artists placed there.
The generative model itself is not imparting structure to the data, it isnt aware of stlyle.. it's sampling from structure that we placed there. When we did so it was because we were, eg., in the room and taking a photo; or imagining what it would be like to apply preraphelite paintaing styles to 60s psychedelic colour pallets because we sensed that fashions of a century ago would now be regarded as cool.
The point of shoving so much data at those models is to help them pick up on the "very very large number of factors".
There was a story I saw on HN a few times in the past, but which I can't find anymore, of someone training a simple, dumb neural net to predict a product (or a sum?) of two numbers, and discovering to their surprise that, under optimization pressure, the network eventually picked up Fourier Transform.
It doesn't seem out of realm of possibility for a large model to pick up on light propagation physics and basic 3D structure of our reality just from watching enough images. After all, the information is implicitly encoded there, and you can handwave a Bayesian argument that it should be extractable.
My undergrad/grad work is in Physics; I presently consult on statistics and AI (and other areas); I may soon start a part-time PhD on how to explain AI models. I am presently, as I type, avoiding rewriting a system to explain AI models because I dislike doing things ive done.
Its quite hard to see the full picture of how these statistical models work without experience across a hard science, stats and AI itself. However, people with backgrounds in mathematical finance would also have enough context. But its seemingly rare in physics, csci, stats, ai, etc. fields alone.
I'd hope that most practitioners in applied statistics could separate properties of the data generating process from properties of its measures; but that hope is fading the more direct experience I have of the field of statistics. I had thought that, at least within the field, you wouldn't have the sort of pseudoscientific thinking that goes along with associative modelling. I think mathematical finance is probably the only area where you can reliably get an end-to-end picture on reality-to-stats models.
Humans have painted with wonky perspective and impossible shadows because they didn't know better for literally 50.000 years. And those humans were just as smart as we are. Just look at 13th century paintings. Does this prove that humans back then didn't understand what a coffee cup looks like when rotated? No. So what does this prove about midjourney? Nothing.
I appreciate then when you're not an expert in physics, statistics and so on, all you have to go on are these circumstantial arguments, "two things that seem similar to me are alike, therefore they are alike in the same way".
However, I am making no such argument. I am explaining that statistical models of pixel frequencies cannot model the causes of those frequencies. I am illustrating this point with an example, not proving it.
If you want more detail about the reason it cannot: when the back of a coffee cup looks like the front, you can generate the back. But you cannot generate the bottom. (assuming the bottom doesn't occur in the dataset) -- why? Because the pixel distributions for the bottom of a cup have zero information about the rest of it.. and the model has no information about the bottom.
If you want a "proof" you'd need at least to be familiar with applied mathematics and the like:
Say the RGB value of each pixel, X of photos of coffee cups obtains from a data generating process parameterized on: distance from camera, lens focal length, angle to cup, lighting conditions, etc. Now produce a model of such causes, call it Environment(distance, angle, cup albedio,...).
Then show that X ~ E|fixed-paramerters induces a frequency distribution of pixels, f1(next|previous) = P(Xi...n|Xj...n); then for any variation in a fixed parameter induces a completely different distribution, say f2, f3, f4, ... Now check that the covariance distribution for most pairs of fs, shows that any given f is almost zero-informative about any other f.
Having done this, compare with a non-statistical (eg., video game) model of Environment where parameters are varied.. and show that all frames, say v, of the video game generated do have high covariance over the time of their sampling. The video game model covaries with most f1..fn; for the associative statistical model it only covaries with f1, or a very small number of others.
There's something very obvious about this if you understand how these statistical AI systems work: in cases where variations in the environment induce radically different distributions the AI will fail; in cases where they are close enough, they will (appear) to succeed.
The marketability of generative AI comes from rigging the use cases to situations where we don't need to change the environment. ie., you aren't exposed to the fact that when you generated a photo you could not have got the same one "at a different distance".
If a video game was built this way it would be unplayable: consider every time you move the camera all the objects randomly change their apparent orientation, distance, style, etc.
Humans have those exact same constraints. For the longest time we could only speculate what the dark side of the moon looked like, for instance.
Yes, LLMs are constrained in what output they can generate based on their training data. Just as we humans are constrained in the output we can generate. When we talk about things we don't understand we speak gibberish, just like LLMs.
Humans have those exact same constraints. For the longest time we could only speculate what the dark side of the moon looked like, for instance.
That isn't the exact same constraint. We could speculate that the moon had a "dark side," because we understood what a moon was, and what a sphere was. LLMs cannot speculate about things outside of their existing data model, at all.
When we talk about things we don't understand we speak gibberish, just like LLMs.
No we don't, wtf? We may create inaccurate models or theories, but we don't just chain together random strings of words the way LLMs do.
Is it so hard to believe that such models have developed a sense for how light propagates through a scene...
This specifically is the thing I usually notice in AI images (outside of the hand trope).
I'm not GP, and at best a layman in the field, but it's not hard to believe it's possible to generate believable lighting, given enough training data, but if I'm not mistaken it would be through sheer volume of properties like lighting/shadow here usually follows item here.
But it's extremely inefficient, and not like we reason. It's like learning the multiplication table without understanding math. Just pairing an infinite amount of properties with each other.
We on the other hand develop a grasp of where lighting exists (sun/lamp) and surmise where shadows fall and can muster any image in our mind using that model instead.
I can go to a huggingface space right now and type in koala wearing a suit serving coffee at a republican rally and there's a reasonable chance I get a result that's something along those lines. Is that meaningfully different to "coffee using a macro lens"?
Those models were not trained on the restricted dataset i'm talking about.
I'm saying you deliberately construct a dataset which, say, does not include cups at various distances, angles, etc. but has as many as you like at a fixed range of these parameters (lens, distance, lighting, angle...).
Now, you will get, from this model, just coffee cup images with these same parameters (eg., distance from the camera).
Real-world generative systems are deliberately not constrained this way, and require many many PBs of images under various conditions to overcome this problem.
Nevertheless you can actually still see this limitation: most generated photos etc. show subjects in "photographic distance/focus/etc. conditions", ie., its hard to get a photo of a person who isnt framed as if they were teh subject of a photo.
Whereas, if you were in a room with a friend, you can take a photo at any angle/distnace.. even, say, from the top of their ears down. You will not get this freedom with a statistical model of pixel patterns
humans tend to agree that in a vast array of cases state of the art LLMs do predict tokens very well
This argument is backwards. Humans don't measure the next token prediction ability of the agents they speak to, human or AI. We rate speakers on whether they seem to understand what we say in context and respond by contributing useful information and analysis.
The attributes you're saying can be inferred from known superior next token prediction ability are the things we can actually detect and measure, at least qualitatively. Next token prediction quality is not measurable by humans in any human-meaningful way. Improving test cross entropy by 50% doesn't mean anything to us. It is irrelevant except as a mechanism to train LLMs.
Point is that the simplest way to excel in next token prediction in the way human consider correct - which is rated by how people feel the predictor mimics a human understanding - is to actually have a world model and other components of human understanding.
Understanding and compression are the same thing. LLMs are fed a huge chunk of totality of human knowledge, and optimized to compress it well. They for sure aren't doing it by Huffman-encoding a multidimensional lookup table.
Point is that the simplest way to excel in next token prediction in the way human consider correct - which is rated by how people feel the predictor mimics a human understanding - is to actually have a world model and other components of human understanding.
This is a speculative theory for why a next token predictor might sound like it knows what it's talking about. Not something we actually know.
In which case you have a trivial point rather than a backwards argument: "the output seems like it knows what's it talking about, and the easiest way to explain that is if it really knows what it's talking about."
Comments
LLMs don’t ’replay variations on prior conversations’ though.
Predicting the probable token in a conversation requires predicting the probable subject of the conversation, predicting the interlocutors’ relationship and manner of speaking to one another, predicting the state of recollection, preference and taste of the speaker, predicting the speaker’s mental model…
If the LLM isn’t predicting all of those things then it will produce poor predictions of the next word; doing it well - and humans tend to agree that in a vast array of cases state of the art LLMs do predict tokens very well - requires that prediction model to predict all that context as well.
Alas, no it doesn't. Language induces this sort of anthropomorphism in people, I guess, so consider images.
Suppose I take a billion images of all the coffee cups in the world, at a set of angles on the cup, and then build an associative (ie., frequency) statistical model of their pixels (ie., statistical AI). Consider generating one pixel at a time, in sequence, through the image. My associative model tells me P(col of next pixel | all previous).
Now, I can generate coffee cups images similar to any variation or combination of the images in the dataset. Now, you might say, "well you can only do that if you have a model of a coffee cup" (rather than of pixels) -- if so, just generate a coffee cup at one of the angles not in the dataset. This will not happen, because the model has not been provided with enough information to do so.
Namely, the model does not know the distance from the camera, the camera lens parameters, the angle to the coffee cup, etc. So there's literally a very very large inifinity of possible objects at unseen angles. Consider that underneath a coffee cup, the bottom might be missing entirely, etc.
Now it will appear to know all of these things, because its just generating images with these same parameters (camera, angle, distance, etc.). But as soon as you want "a coffee further away than has been seen before", or "a coffee using a macro lens", etc. the whole thing will fall over.
It is you, the view, who attributes 3D knowledge to the model because under ordinary circumstances the cause of a photo is features of a 3D environment.
You’re saying this with confidence as if there isn’t a large body of working image and video generation algorithms out there that can produce physically plausible images of objects transposed into circumstances that don’t exist in their training set. A coffee using a macro lens for example.
Is it so hard to believe that such models have developed a sense for how light propagates through a scene, a sense for how physical objects change when viewed from different angles, a sense for how lens distortion interacts with light? For goodness’ sake, these same models have a sense of what Greg Rutkowski’s art style is - we are well beyond ‘they’re just remembering pixels from past coffeecups’
Well, its not a matter or belief or otherwise. I'm a trained practitioner in statistics, AI, physics, and other areas and you can show trivially that you cannot learn light physics from pixel distributions.
Pixel distributions aren't stationary, and are caused by a very very large number of factors; likewise the physics of light for any given situation is subject to a large number of causes, all of them entirely absent from from the pixel distributions. This is a pretty trivial thing to show.
Well what these models show is that when you have PBs of image data and TBs of associated text data, you can relate words and images together usefully. In particular, you can use patterns of text tokens to sample from image distributions, and combine and vary these samples to produce novel images.
The patterns in text and images are caused by people speaking, taking photos, etc. Those patterns necessarily obtain in any generated output. As in, if you train an LLM/etc. on how to speak, using vast amounts of conversational data, it cannot do anything other than appear to speak: that is the only thing the data distribution makes possible.
Likewise here, the image generator has a compressed representation of PBs of pixel data which can be sampled from using text. So when you say, "Greg Rutkowski" you select for a highly structured image space, whose structure the original artists placed there.
The generative model itself is not imparting structure to the data, it isnt aware of stlyle.. it's sampling from structure that we placed there. When we did so it was because we were, eg., in the room and taking a photo; or imagining what it would be like to apply preraphelite paintaing styles to 60s psychedelic colour pallets because we sensed that fashions of a century ago would now be regarded as cool.
The point of shoving so much data at those models is to help them pick up on the "very very large number of factors".
There was a story I saw on HN a few times in the past, but which I can't find anymore, of someone training a simple, dumb neural net to predict a product (or a sum?) of two numbers, and discovering to their surprise that, under optimization pressure, the network eventually picked up Fourier Transform.
It doesn't seem out of realm of possibility for a large model to pick up on light propagation physics and basic 3D structure of our reality just from watching enough images. After all, the information is implicitly encoded there, and you can handwave a Bayesian argument that it should be extractable.
Genuine question, what does it mean to be a trained practitioner in statistics, AI, physics and other areas?
My undergrad/grad work is in Physics; I presently consult on statistics and AI (and other areas); I may soon start a part-time PhD on how to explain AI models. I am presently, as I type, avoiding rewriting a system to explain AI models because I dislike doing things ive done.
Its quite hard to see the full picture of how these statistical models work without experience across a hard science, stats and AI itself. However, people with backgrounds in mathematical finance would also have enough context. But its seemingly rare in physics, csci, stats, ai, etc. fields alone.
I'd hope that most practitioners in applied statistics could separate properties of the data generating process from properties of its measures; but that hope is fading the more direct experience I have of the field of statistics. I had thought that, at least within the field, you wouldn't have the sort of pseudoscientific thinking that goes along with associative modelling. I think mathematical finance is probably the only area where you can reliably get an end-to-end picture on reality-to-stats models.
Humans have painted with wonky perspective and impossible shadows because they didn't know better for literally 50.000 years. And those humans were just as smart as we are. Just look at 13th century paintings. Does this prove that humans back then didn't understand what a coffee cup looks like when rotated? No. So what does this prove about midjourney? Nothing.
I appreciate then when you're not an expert in physics, statistics and so on, all you have to go on are these circumstantial arguments, "two things that seem similar to me are alike, therefore they are alike in the same way".
However, I am making no such argument. I am explaining that statistical models of pixel frequencies cannot model the causes of those frequencies. I am illustrating this point with an example, not proving it.
If you want more detail about the reason it cannot: when the back of a coffee cup looks like the front, you can generate the back. But you cannot generate the bottom. (assuming the bottom doesn't occur in the dataset) -- why? Because the pixel distributions for the bottom of a cup have zero information about the rest of it.. and the model has no information about the bottom.
If you want a "proof" you'd need at least to be familiar with applied mathematics and the like:
Say the RGB value of each pixel, X of photos of coffee cups obtains from a data generating process parameterized on: distance from camera, lens focal length, angle to cup, lighting conditions, etc. Now produce a model of such causes, call it Environment(distance, angle, cup albedio,...).
Then show that X ~ E|fixed-paramerters induces a frequency distribution of pixels, f1(next|previous) = P(Xi...n|Xj...n); then for any variation in a fixed parameter induces a completely different distribution, say f2, f3, f4, ... Now check that the covariance distribution for most pairs of fs, shows that any given f is almost zero-informative about any other f.
Having done this, compare with a non-statistical (eg., video game) model of Environment where parameters are varied.. and show that all frames, say v, of the video game generated do have high covariance over the time of their sampling. The video game model covaries with most f1..fn; for the associative statistical model it only covaries with f1, or a very small number of others.
There's something very obvious about this if you understand how these statistical AI systems work: in cases where variations in the environment induce radically different distributions the AI will fail; in cases where they are close enough, they will (appear) to succeed.
The marketability of generative AI comes from rigging the use cases to situations where we don't need to change the environment. ie., you aren't exposed to the fact that when you generated a photo you could not have got the same one "at a different distance".
If a video game was built this way it would be unplayable: consider every time you move the camera all the objects randomly change their apparent orientation, distance, style, etc.
Humans have those exact same constraints. For the longest time we could only speculate what the dark side of the moon looked like, for instance.
Yes, LLMs are constrained in what output they can generate based on their training data. Just as we humans are constrained in the output we can generate. When we talk about things we don't understand we speak gibberish, just like LLMs.
That isn't the exact same constraint. We could speculate that the moon had a "dark side," because we understood what a moon was, and what a sphere was. LLMs cannot speculate about things outside of their existing data model, at all.
No we don't, wtf? We may create inaccurate models or theories, but we don't just chain together random strings of words the way LLMs do.
This specifically is the thing I usually notice in AI images (outside of the hand trope).
I'm not GP, and at best a layman in the field, but it's not hard to believe it's possible to generate believable lighting, given enough training data, but if I'm not mistaken it would be through sheer volume of properties like lighting/shadow here usually follows item here.
But it's extremely inefficient, and not like we reason. It's like learning the multiplication table without understanding math. Just pairing an infinite amount of properties with each other.
We on the other hand develop a grasp of where lighting exists (sun/lamp) and surmise where shadows fall and can muster any image in our mind using that model instead.
Is that really true?
I can go to a huggingface space right now and type in koala wearing a suit serving coffee at a republican rally and there's a reasonable chance I get a result that's something along those lines. Is that meaningfully different to "coffee using a macro lens"?
Those models were not trained on the restricted dataset i'm talking about.
I'm saying you deliberately construct a dataset which, say, does not include cups at various distances, angles, etc. but has as many as you like at a fixed range of these parameters (lens, distance, lighting, angle...).
Now, you will get, from this model, just coffee cup images with these same parameters (eg., distance from the camera).
Real-world generative systems are deliberately not constrained this way, and require many many PBs of images under various conditions to overcome this problem.
Nevertheless you can actually still see this limitation: most generated photos etc. show subjects in "photographic distance/focus/etc. conditions", ie., its hard to get a photo of a person who isnt framed as if they were teh subject of a photo.
Whereas, if you were in a room with a friend, you can take a photo at any angle/distnace.. even, say, from the top of their ears down. You will not get this freedom with a statistical model of pixel patterns
I can't argue with that, so I think unfortunately I may have missed the original point.
The sun revolved around the earth for a long time until our own model was updated to include more data.
This argument is backwards. Humans don't measure the next token prediction ability of the agents they speak to, human or AI. We rate speakers on whether they seem to understand what we say in context and respond by contributing useful information and analysis.
The attributes you're saying can be inferred from known superior next token prediction ability are the things we can actually detect and measure, at least qualitatively. Next token prediction quality is not measurable by humans in any human-meaningful way. Improving test cross entropy by 50% doesn't mean anything to us. It is irrelevant except as a mechanism to train LLMs.
Point is that the simplest way to excel in next token prediction in the way human consider correct - which is rated by how people feel the predictor mimics a human understanding - is to actually have a world model and other components of human understanding.
Understanding and compression are the same thing. LLMs are fed a huge chunk of totality of human knowledge, and optimized to compress it well. They for sure aren't doing it by Huffman-encoding a multidimensional lookup table.
This is a speculative theory for why a next token predictor might sound like it knows what it's talking about. Not something we actually know.
I mean, I think it was implied that humans judge the ‘next token prediction’ ability of LLMs as being good based on the quality of the overall output.
In which case you have a trivial point rather than a backwards argument: "the output seems like it knows what's it talking about, and the easiest way to explain that is if it really knows what it's talking about."