I don't know the answer to this, either, of course. :) I have a strong suspicion that LLMs are not, in and of themselves, the way AGI will be achieved. They may be a component of it, but I suspect there is a fundamental difference between "based on knowledge and context, I am confident this is a correct answer to your question" and "based on statistical and lexical analysis, I am confident this is what a correct answer to your question might look like," and I don't think LLM-only technology isn't capable of making the jump from the latter to the former.
So is just getting better and better at the latter enough? In some contexts, probably -- but somewhat ironically, I don't think those are the contexts that we keep pushing AI into. It's pretty good with problem spaces for which "correctness" isn't a terribly important metric. For instance, I just prompted ChatGPT 3.5-turbo to come up with a synopsis of a 1920s private detective story, with a few provided details, using the seven-point story structure, and I was surprised at how well it did. It came up with something really derivative, sure (to the point where it named its protagonist "Jack Malone," which sounds suitably detective-ish because Jack Malone turns out to be the main character of the long-running detective show "Without a Trace"), but definitely a passing grade!
But time and time and time again we see examples of AI failing at areas where correctness is important -- "hallucinations" -- and I think that's where my distinction in the first paragraph becomes more than just mere semantics. LLMs do not actually have knowledge and context, but instead just generate something that looks correct. Sometimes it actually is correct! (I could ask ChatGPT to summarize the seven-point story structure and it aced it, for instance.) But sometimes it isn't, and the LLM literally cannot distinguish between correct and incorrect responses. I ran into this at my last job, where…hmm. To be a bit elliptical about it to avoid breaking an NDA I suspect I'm still under, we were trying to train an LLM to call a JSON-based API by analyzing the user's query and filling in the correct API parameters. It did an amazing job most of the time, but the remit was "if the user doesn't specify a required parameter, prompt them for it," and there was no way to get the LLM to do that consistently. Sometimes it would, but sometimes it would just make up what it thought was a plausible value and plunge ahead. "Works nine times out of ten" may or may not be an acceptable success rate, but "one time out of ten makes up shit and sees what happens" is not an acceptable failure mode.
And this is where Chiang's thermostat analogy comes into play for me, I think: I simply don't see how you get from "looks correct" to "verifiably correct" by adding more and better thermostats. You're going to get responses that sound better, and you're going to get them faster, but you're still not going to be able to know whether they're bullshit without running them by a fact-checker.
Comments
I don't know the answer to this, either, of course. :) I have a strong suspicion that LLMs are not, in and of themselves, the way AGI will be achieved. They may be a component of it, but I suspect there is a fundamental difference between "based on knowledge and context, I am confident this is a correct answer to your question" and "based on statistical and lexical analysis, I am confident this is what a correct answer to your question might look like," and I don't think LLM-only technology isn't capable of making the jump from the latter to the former.
So is just getting better and better at the latter enough? In some contexts, probably -- but somewhat ironically, I don't think those are the contexts that we keep pushing AI into. It's pretty good with problem spaces for which "correctness" isn't a terribly important metric. For instance, I just prompted ChatGPT 3.5-turbo to come up with a synopsis of a 1920s private detective story, with a few provided details, using the seven-point story structure, and I was surprised at how well it did. It came up with something really derivative, sure (to the point where it named its protagonist "Jack Malone," which sounds suitably detective-ish because Jack Malone turns out to be the main character of the long-running detective show "Without a Trace"), but definitely a passing grade!
But time and time and time again we see examples of AI failing at areas where correctness is important -- "hallucinations" -- and I think that's where my distinction in the first paragraph becomes more than just mere semantics. LLMs do not actually have knowledge and context, but instead just generate something that looks correct. Sometimes it actually is correct! (I could ask ChatGPT to summarize the seven-point story structure and it aced it, for instance.) But sometimes it isn't, and the LLM literally cannot distinguish between correct and incorrect responses. I ran into this at my last job, where…hmm. To be a bit elliptical about it to avoid breaking an NDA I suspect I'm still under, we were trying to train an LLM to call a JSON-based API by analyzing the user's query and filling in the correct API parameters. It did an amazing job most of the time, but the remit was "if the user doesn't specify a required parameter, prompt them for it," and there was no way to get the LLM to do that consistently. Sometimes it would, but sometimes it would just make up what it thought was a plausible value and plunge ahead. "Works nine times out of ten" may or may not be an acceptable success rate, but "one time out of ten makes up shit and sees what happens" is not an acceptable failure mode.
And this is where Chiang's thermostat analogy comes into play for me, I think: I simply don't see how you get from "looks correct" to "verifiably correct" by adding more and better thermostats. You're going to get responses that sound better, and you're going to get them faster, but you're still not going to be able to know whether they're bullshit without running them by a fact-checker.