convincing simulations of conscious thought entirely via predicting language tokens means something profound about the nature of language itself.
there's also something big about us waiting there under the surface.
I don't believe so. In "The Origins of Knowledge and Imagination" by Jacob Brownoski, he argues that human language have four unique characteristics:
- We can separate information (data of what being described) from emotional content (how we're supposed to react). There's no longer a bijection between communication and action.
- We can extend the time reference of the communication content. We talk about the past, we plan for the future.
- We can refer to ourselves. So we examine what we've done and iterate over it until we fix the errors. We can see ourselves doing the action without actually doing it.
- We can rearrange units of languages to have different meanings. The same words can have different meanings based on their order. So meaning depends not only on the words, but their sequence. And that goes from words to phrases to sequence of dialogs.
The fourth point is the most important. LLMs by predicting languages tokens can give use the most common order for a particular context. And because we don't have that many words, their orders can be extracted from books and other written content. But then they fail for the higher levels, mostly because that's when everything get unique.
As for the third point, by observing ourselves, our communication is constantly being based on reality, which grounds it in truth. And because we can extend the reference it's based on, that leads us to observe changes and model laws. The first point allows us to separate what things are from what we should do or feel based on their existence and absence.
Instead of the LLMs fooling us, it's more us fooling ourselves, because by recognizing meaning in sentences, we try to extract meanings for longer sequences of text where there aren't any. Why? Because there is no "I" that has done the job of extracting information and using language to transmit it (while still cognizant of the imperfection of natural languages). LLMs are lossy compressions of ideas. Only the smallest survives and then it generates much more false ones.
Are you certain that you're not playing with words to arrive at a predetermined conclusion? What is this "I" to which you're referring and how can you demonstrate that "I" does not or cannot exist within systems such as these? Further, if you are to find something which qualifies as an "I" elsewhere, what makes that elsewhere fundamentally different and therefore capable of supporting and being an "I" and is that elsewhere such simply by definition or in and of itself? Further, if the language usage is indistinguishable from the language usage of an "I", is the difference of source meaningful? If so, why?
The "I" is stemmed from the theory of the mind. We can only access our own mind and thus has no way to infer the thoughts of other. So we observe them and infer based on our own patterns. In a sense, we assume that others have the same mechanism that we possess, and thus we engage in interactions with them. So far, there is no demonstration of reasoning within systems such as these, it's all simulation of the communication channel themselves.
Further, if the language usage is indistinguishable from the language usage of an "I", is the difference of source meaningful?
Is it indistinguishable? The first thing we look for in communication is consistency so that we can examine for intent. And this is after we determine the other party. Because we know the intent is not ours. But what I've seen of prompt engineering is that the communication intent always come from the person, not the models. Then it goes on to find the most likely continuation of this intent (based on the model training) and then it quickly become an echo chamber. It's search in lexical space and you can see the limits when it became a oscillating loop between the same set of reply. Because there's no "I don't know" damping.
Why does there need to be an "I" that uses language to transmit information? Language itself encodes information. I can read a piece of text and gain something from it. Where the text came from is irrelevant.
Which it does in a lossy manner. Information is independent from language. The more complex the information, the more language fails. Which is why there is so many mediums for communication. Language has three main components: the symbols, the grammar, and the dictionary. The first refers to the tokens of our vocabulary, the second to the rules to arrange these tokens, and the third describes the relation of the tokens to the things they represent.
The relation between the three is interdependent. We name new things we encounter, creating entry in the dictionary, we figure the rules that governs these things, and the relation to other things encountered previously. And thus, we can issue statements. We can also name these statements and it continues recursively. But each of us possess its own copy of these stuff with its own variations. What you gain from what I said may be different from what I intended to transmit. And what I intended to transmit may be a poor description of the things itself. So flawed interpretation, flawed description, and flawed transmission result in flawed understanding. To correct it, you need to be in presence of the thing itself. Missing that, you strive to establish the tokens, the grammar, and the dictionary of the person that have written the text.
In LLMs, the dictionary is missing. The token "snow" has no relation to the thing we call snow. But because it's often placed near other tokens like "ice", "freeze", etc,... Then a rule emerges (embedding?) that these things must be related to each other. In what way it does not know. But if we apply the data collected in a statistical manner, we can arrange these tokens and the result will probably be correct. But there's still a non-zero chance that the generated statement is meaningless as there's no foundation rule that drives it. So there's only tokens. And rules derived from analyzing texts (which lack the foundation rules that comes from being in the real world).
All of these to say the act of learning is either observing the real world and figure how it works. Or read from someone that has done the observing and has written his interpretation, then go outside and confirm it. Barring that, we reconstruct the life of this person so that we can correct the imperfection of languages. With LLMs, there's no way to correct as the statement themselves are not truthful. they can just be accidentally be right.
I think the core insight OP may be looking for is that your dictionary is just an illusion - that concepts being related to other concepts to various degree is all that there is. The meaning of a concept is defined entirely by other concepts that are close to it in something like a latent space of a language model.
Of course humans get to also connect concepts with inputs from other senses, such as sight, touch, smell or sound. This provides some grounding. It is important for learning to communicate (and to have something to communicate about), and was important for humans when first developing languages - but they're not strictly necessary to learn the meanings. All this empirical grounding is already implicitly encoded in human communication, so it should be possible for an LLM to actually understand what e.g. "green" means, despite having never seen color. Case in point: blind people are able to do this, so the information is there.
Blind people are no more able to understand* (as qualia) "green" than a sighted human is able to understand* gamma rays. The confusion is between working with abstract concepts vs an actual experience. A picture of bread provides no physical nourishment beyond the fiber in the paper it is printed on.
In an abstract space (e.g. word vectors, poetry) green could have (many potential) meanings. But none of them are even in the same universe as the actual experience (qualia) of seeing something green. This would be a category mistake between qualia-space and concept-space
I don't need the qualia of gamma rays to understand gamma rays, nor to be understood in turn when I say that "I understand gamma rays".
Conversely, I can (and do) have qualia that I do not understand.
The concept of qualia is, I think, pre-paradigmatic — we know of our own, but can't turn that experience into a testable phenomena in the world outside our heads. We don't have any way to know if any given AI does or doesn't have it, nor how that might change as the models go from text to multimodal, or if we give them (real or simulated) embodiment.
Comments
I don't believe so. In "The Origins of Knowledge and Imagination" by Jacob Brownoski, he argues that human language have four unique characteristics:
- We can separate information (data of what being described) from emotional content (how we're supposed to react). There's no longer a bijection between communication and action.
- We can extend the time reference of the communication content. We talk about the past, we plan for the future.
- We can refer to ourselves. So we examine what we've done and iterate over it until we fix the errors. We can see ourselves doing the action without actually doing it.
- We can rearrange units of languages to have different meanings. The same words can have different meanings based on their order. So meaning depends not only on the words, but their sequence. And that goes from words to phrases to sequence of dialogs.
The fourth point is the most important. LLMs by predicting languages tokens can give use the most common order for a particular context. And because we don't have that many words, their orders can be extracted from books and other written content. But then they fail for the higher levels, mostly because that's when everything get unique.
As for the third point, by observing ourselves, our communication is constantly being based on reality, which grounds it in truth. And because we can extend the reference it's based on, that leads us to observe changes and model laws. The first point allows us to separate what things are from what we should do or feel based on their existence and absence.
Instead of the LLMs fooling us, it's more us fooling ourselves, because by recognizing meaning in sentences, we try to extract meanings for longer sequences of text where there aren't any. Why? Because there is no "I" that has done the job of extracting information and using language to transmit it (while still cognizant of the imperfection of natural languages). LLMs are lossy compressions of ideas. Only the smallest survives and then it generates much more false ones.
Are you certain that you're not playing with words to arrive at a predetermined conclusion? What is this "I" to which you're referring and how can you demonstrate that "I" does not or cannot exist within systems such as these? Further, if you are to find something which qualifies as an "I" elsewhere, what makes that elsewhere fundamentally different and therefore capable of supporting and being an "I" and is that elsewhere such simply by definition or in and of itself? Further, if the language usage is indistinguishable from the language usage of an "I", is the difference of source meaningful? If so, why?
The "I" is stemmed from the theory of the mind. We can only access our own mind and thus has no way to infer the thoughts of other. So we observe them and infer based on our own patterns. In a sense, we assume that others have the same mechanism that we possess, and thus we engage in interactions with them. So far, there is no demonstration of reasoning within systems such as these, it's all simulation of the communication channel themselves.
Is it indistinguishable? The first thing we look for in communication is consistency so that we can examine for intent. And this is after we determine the other party. Because we know the intent is not ours. But what I've seen of prompt engineering is that the communication intent always come from the person, not the models. Then it goes on to find the most likely continuation of this intent (based on the model training) and then it quickly become an echo chamber. It's search in lexical space and you can see the limits when it became a oscillating loop between the same set of reply. Because there's no "I don't know" damping.
Why does there need to be an "I" that uses language to transmit information? Language itself encodes information. I can read a piece of text and gain something from it. Where the text came from is irrelevant.
Which it does in a lossy manner. Information is independent from language. The more complex the information, the more language fails. Which is why there is so many mediums for communication. Language has three main components: the symbols, the grammar, and the dictionary. The first refers to the tokens of our vocabulary, the second to the rules to arrange these tokens, and the third describes the relation of the tokens to the things they represent.
The relation between the three is interdependent. We name new things we encounter, creating entry in the dictionary, we figure the rules that governs these things, and the relation to other things encountered previously. And thus, we can issue statements. We can also name these statements and it continues recursively. But each of us possess its own copy of these stuff with its own variations. What you gain from what I said may be different from what I intended to transmit. And what I intended to transmit may be a poor description of the things itself. So flawed interpretation, flawed description, and flawed transmission result in flawed understanding. To correct it, you need to be in presence of the thing itself. Missing that, you strive to establish the tokens, the grammar, and the dictionary of the person that have written the text.
In LLMs, the dictionary is missing. The token "snow" has no relation to the thing we call snow. But because it's often placed near other tokens like "ice", "freeze", etc,... Then a rule emerges (embedding?) that these things must be related to each other. In what way it does not know. But if we apply the data collected in a statistical manner, we can arrange these tokens and the result will probably be correct. But there's still a non-zero chance that the generated statement is meaningless as there's no foundation rule that drives it. So there's only tokens. And rules derived from analyzing texts (which lack the foundation rules that comes from being in the real world).
All of these to say the act of learning is either observing the real world and figure how it works. Or read from someone that has done the observing and has written his interpretation, then go outside and confirm it. Barring that, we reconstruct the life of this person so that we can correct the imperfection of languages. With LLMs, there's no way to correct as the statement themselves are not truthful. they can just be accidentally be right.
I think the core insight OP may be looking for is that your dictionary is just an illusion - that concepts being related to other concepts to various degree is all that there is. The meaning of a concept is defined entirely by other concepts that are close to it in something like a latent space of a language model.
Of course humans get to also connect concepts with inputs from other senses, such as sight, touch, smell or sound. This provides some grounding. It is important for learning to communicate (and to have something to communicate about), and was important for humans when first developing languages - but they're not strictly necessary to learn the meanings. All this empirical grounding is already implicitly encoded in human communication, so it should be possible for an LLM to actually understand what e.g. "green" means, despite having never seen color. Case in point: blind people are able to do this, so the information is there.
Blind people are no more able to understand* (as qualia) "green" than a sighted human is able to understand* gamma rays. The confusion is between working with abstract concepts vs an actual experience. A picture of bread provides no physical nourishment beyond the fiber in the paper it is printed on.
In an abstract space (e.g. word vectors, poetry) green could have (many potential) meanings. But none of them are even in the same universe as the actual experience (qualia) of seeing something green. This would be a category mistake between qualia-space and concept-space
understand in the experiential, qualia sense.
https://en.wikipedia.org/wiki/Qualia
https://en.wikipedia.org/wiki/Category_mistake
I don't need the qualia of gamma rays to understand gamma rays, nor to be understood in turn when I say that "I understand gamma rays".
Conversely, I can (and do) have qualia that I do not understand.
The concept of qualia is, I think, pre-paradigmatic — we know of our own, but can't turn that experience into a testable phenomena in the world outside our heads. We don't have any way to know if any given AI does or doesn't have it, nor how that might change as the models go from text to multimodal, or if we give them (real or simulated) embodiment.
This is the view that Fodor termed "inferential role semantics". https://ruccs.rutgers.edu/images/personal-ernest-lepore/WhyM...