I got into the habit of running every query I do through all of the SOTA models so I can directly compare the results.
This was originally GPT-4, Claude 3 Opus, and Gemini Advanced. I recently added Meta AI when they launched.
Right now I've sent 486 queries through the first three systems.
The clearest pattern to emerge is that Gemini is terrible, not on par with the other two. There hasn't been a single query that it was the only model who did well. Around 1/4 of the time it gives a clearly inferior answer to the others.
But between GPT-4 and Claude it's less clear. 31 of the 486 queries Claude provided a significantly better answer than the other two but 20 times GPT-4 provided the significantly better answer.
I do think that Claude is a slightly better model but right now it's not a clear enough advantage that I'd recommend it generally. I will say you can probably cancel you Gemini subscription if you're using it though.
One thing I would say about Opus over ChatGPT is that it can return code without placeholders whereas ChatGPT will always have placeholders. It wasn't always that way though and it seems like cutting corners.
You might want to check out perplexity.AI. It has access to many of these models, and it’s trivial to switch models mid conversation and ask it to repeat itself, so that you can see the differences.
Why isn’t the model offered by Google in the same category? Or did I misunderstand their latest naming fiasco and “Gemini Pro” is not actually their latest model?
EDIT: It looks like Gemini Ultra 1.0 > Gemini Pro 1.5 > Gemini Pro 1.0… is that true? What a horrible versioning scheme.
Comments
I got into the habit of running every query I do through all of the SOTA models so I can directly compare the results.
This was originally GPT-4, Claude 3 Opus, and Gemini Advanced. I recently added Meta AI when they launched.
Right now I've sent 486 queries through the first three systems.
The clearest pattern to emerge is that Gemini is terrible, not on par with the other two. There hasn't been a single query that it was the only model who did well. Around 1/4 of the time it gives a clearly inferior answer to the others.
But between GPT-4 and Claude it's less clear. 31 of the 486 queries Claude provided a significantly better answer than the other two but 20 times GPT-4 provided the significantly better answer.
I do think that Claude is a slightly better model but right now it's not a clear enough advantage that I'd recommend it generally. I will say you can probably cancel you Gemini subscription if you're using it though.
For everyday non-dev users, Msty for MacOS lets you run split chats and compare each model's answers.
https://msty.app
One thing I would say about Opus over ChatGPT is that it can return code without placeholders whereas ChatGPT will always have placeholders. It wasn't always that way though and it seems like cutting corners.
interesting, this was a problem around the turn of the year for me but chatgpt has stopped doing it now
You might want to check out perplexity.AI. It has access to many of these models, and it’s trivial to switch models mid conversation and ask it to repeat itself, so that you can see the differences.
Why would you compare Gemini pro with the other two - they are not in the same category. Did you mean to say advanced?
Why isn’t the model offered by Google in the same category? Or did I misunderstand their latest naming fiasco and “Gemini Pro” is not actually their latest model?
EDIT: It looks like Gemini Ultra 1.0 > Gemini Pro 1.5 > Gemini Pro 1.0… is that true? What a horrible versioning scheme.
Yes I do mean Gemini Advanced.