So it appears that GPT-4-Turbo is indeed (at least marginally) smarter than the previous GPT-4, just as Altman claimed. Also, it's faster and cheaper, with a massive context window. Exciting!
I haven't tried it yet, but people in the /r/chatgpt subreddit are claiming GPT-4-Turbo seems to have issues with understanding/remembering longer (say 100 lines) of code, whereas 3.5 and 4.0 seem to have handled things a bit better, implying that the context-window size isn't (currently) as large as claimed.
The context window IS longer, but it's less powerful. Obviously, they can't afford to have full transformer context over the entire context. That would be an impossibly large amount of ram. They're using some combination of sliding window/cyclical/or some other adjusted attention mechanism likely with some degree of summarization in some manner.
The increased context size will have the most significant impact on my work. That's where I run into limitations, when reviewing written work and code. I've been feeding written work into GPT in chunks, and I'm really happy to be able to feed in whole pieces. (I don't have it revise anything for me, I just have a specific prompt for exactly the kind of feedback I want on written work.)
I tried Claude because of the larger context size, but I've been disappointed so far. I find Claude much more likely to just compliment my writing, whereas GPT will identify strengths and areas that could be improved.
Have you tried aider-chat? It does some interesting things with tree-sitter so it can give the LLM a context (files, classes, functions, parameters, etc) as well as certain full files. That way it has your entire codebase in API form and it can focus on the actual code you are looking at or editing.
Yes I was really worried this was going to be "better" as in faster and cheaper but not "better" as in smarter. I've been playing with it for the past day and haven't noticed it any smarter per se, but also haven't noticed it dummer either.
Perhaps it may have worse reasoning for some tasks, so having "turbo" lets people know they can still try the old version if turbo doesn't work for them? Kind of like 3.5 vs 3.5 Turbo.
Comments
So it appears that GPT-4-Turbo is indeed (at least marginally) smarter than the previous GPT-4, just as Altman claimed. Also, it's faster and cheaper, with a massive context window. Exciting!
I haven't tried it yet, but people in the /r/chatgpt subreddit are claiming GPT-4-Turbo seems to have issues with understanding/remembering longer (say 100 lines) of code, whereas 3.5 and 4.0 seem to have handled things a bit better, implying that the context-window size isn't (currently) as large as claimed.
Anyone else seeing any evidence of this?
Some specialized subreddits can be incredibly useful. /r/chatgpt due to its popularity is not one of those.
It's full of memes and people complaining its not as "good" as it was yesterday when it fails at completing their homework.
I would take anything said there with a big grain of salt, and stick to benchmarks.
The context window IS longer, but it's less powerful. Obviously, they can't afford to have full transformer context over the entire context. That would be an impossibly large amount of ram. They're using some combination of sliding window/cyclical/or some other adjusted attention mechanism likely with some degree of summarization in some manner.
The increased context size will have the most significant impact on my work. That's where I run into limitations, when reviewing written work and code. I've been feeding written work into GPT in chunks, and I'm really happy to be able to feed in whole pieces. (I don't have it revise anything for me, I just have a specific prompt for exactly the kind of feedback I want on written work.)
I tried Claude because of the larger context size, but I've been disappointed so far. I find Claude much more likely to just compliment my writing, whereas GPT will identify strengths and areas that could be improved.
Have you tried aider-chat? It does some interesting things with tree-sitter so it can give the LLM a context (files, classes, functions, parameters, etc) as well as certain full files. That way it has your entire codebase in API form and it can focus on the actual code you are looking at or editing.
Yes I was really worried this was going to be "better" as in faster and cheaper but not "better" as in smarter. I've been playing with it for the past day and haven't noticed it any smarter per se, but also haven't noticed it dummer either.
It's a big surprise. I assumed something would have to have been "lobotomized" to make this speed increase. I'd love to know what they did.
Well, the knowledge cutoff is much more recent (I think sama said April 2023?), so having more, newer data might be a significant contributing factor.
If this is the case, why use the word turbo in the name which has the baggage of faster but worse reasoning.
Perhaps it may have worse reasoning for some tasks, so having "turbo" lets people know they can still try the old version if turbo doesn't work for them? Kind of like 3.5 vs 3.5 Turbo.