You'd think that LLMs fulfill this, but I do wonder if a clever human could still discern between them given their particularities.
I think the clear winning strategy is to find some of the common sense reasoning fails that are still quite common, like the "should I walk or drive to the car wash" one from a few months ago.
These are usually from a missing context (i.e. specifying where the car is right now). Tokenizer issues (how many Xs are in Y) are a better giveaway, but models can solve them using tools nowadays. I think the best way to uncover an LLM is to reveal its superhuman aspects. Ask it for an A* algorithm for your problem. If it can just type down a working algorithm in a few seconds, that is definitely not a human. Although with some careful system prompt design, you could probably mask that aspect too.
These are usually from a missing context (i.e. specifying where the car is right now).
That shouldn't be an issue. The very act of taking the car to car wash contains the information that it is with you and you are at your home. Doesn't this prove that LLMs don't actually think?
What if your car is not brought to the wash by you (common for companies or rich people). If someone asks you how to get it and the car wash is close by, of course you will tell them to go on foot. Human life experience is not universal and thus context can never be either. This is also a huge problem in inter-human communication, since most people suck at putting themselves in other people's shoes. But for LLMs everyone somehow assumes the humans must be right by default, when in fact you can tell even less about a model's training and world experience than for an average human.
These are usually from a missing context (i.e. specifying where the car is right now).
The reason for the mistake is irrelevant here. All that matters is that you catch the agent saying something that basically no human would, especially in this type of game.
No, the context is obvious to any human, that's the whole point. No one would think you've left your car at a car wash, as no one ever does that, and if they did, they wouldn't even consider driving there. It's obvious to any human that the question is a little trick and that the answer is extremely obvious.
Counter example: A person who always has their car brought to the car wash by someone else (could be super wealthy, could be company cars, could be a million other reasons...) is asked whether to go or drive to the car wash. Of course they'll tell you to just walk if it's close anyway. To them the car is already there implicitly. Human life experience is not universal. And so context can't be either.
In a true competitive Turing test, these reasoning pitfalls could be included in the system prompt, there are not that many of them. The tester would need to find a novel reasoning pitfall, but those aren't uncommon among humans either, so that would not make a very useful differentiator.
Another approach could be to probe the helpfulness of an AI. A human would likely not comply with a request to write a sonnet about tangerines.
I don't understand what you mean that "there are not that many of them". The models, even the best ones, still have clear problems with certain kinds of real world reasoning, and it's not a known, closed set. With some prodding and knowing this error mode, I think it's not hard to trip any model with a novel reasoning error like this.
And no, I don't think humans would make the same errors, the key reason for this failure mode is that there are some facts of human day to day life that are so mundane that they are virtually never written about, thus making them an ideal combination of universal human experiences but out of distribution for the LLMs.
Comments
I think the clear winning strategy is to find some of the common sense reasoning fails that are still quite common, like the "should I walk or drive to the car wash" one from a few months ago.
These are usually from a missing context (i.e. specifying where the car is right now). Tokenizer issues (how many Xs are in Y) are a better giveaway, but models can solve them using tools nowadays. I think the best way to uncover an LLM is to reveal its superhuman aspects. Ask it for an A* algorithm for your problem. If it can just type down a working algorithm in a few seconds, that is definitely not a human. Although with some careful system prompt design, you could probably mask that aspect too.
That shouldn't be an issue. The very act of taking the car to car wash contains the information that it is with you and you are at your home. Doesn't this prove that LLMs don't actually think?
What if your car is not brought to the wash by you (common for companies or rich people). If someone asks you how to get it and the car wash is close by, of course you will tell them to go on foot. Human life experience is not universal and thus context can never be either. This is also a huge problem in inter-human communication, since most people suck at putting themselves in other people's shoes. But for LLMs everyone somehow assumes the humans must be right by default, when in fact you can tell even less about a model's training and world experience than for an average human.
The reason for the mistake is irrelevant here. All that matters is that you catch the agent saying something that basically no human would, especially in this type of game.
If it is missing context, it will be indistinguishable from mistakes that humans could make as well.
No, the context is obvious to any human, that's the whole point. No one would think you've left your car at a car wash, as no one ever does that, and if they did, they wouldn't even consider driving there. It's obvious to any human that the question is a little trick and that the answer is extremely obvious.
Counter example: A person who always has their car brought to the car wash by someone else (could be super wealthy, could be company cars, could be a million other reasons...) is asked whether to go or drive to the car wash. Of course they'll tell you to just walk if it's close anyway. To them the car is already there implicitly. Human life experience is not universal. And so context can't be either.
In a true competitive Turing test, these reasoning pitfalls could be included in the system prompt, there are not that many of them. The tester would need to find a novel reasoning pitfall, but those aren't uncommon among humans either, so that would not make a very useful differentiator.
Another approach could be to probe the helpfulness of an AI. A human would likely not comply with a request to write a sonnet about tangerines.
I don't understand what you mean that "there are not that many of them". The models, even the best ones, still have clear problems with certain kinds of real world reasoning, and it's not a known, closed set. With some prodding and knowing this error mode, I think it's not hard to trip any model with a novel reasoning error like this.
And no, I don't think humans would make the same errors, the key reason for this failure mode is that there are some facts of human day to day life that are so mundane that they are virtually never written about, thus making them an ideal combination of universal human experiences but out of distribution for the LLMs.