Comment on From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]Comments−username1351yI wonder what changed with the models that created regression?−teaearlgraycold1yNot sure but with each release it feels like they’re just wiping the dirt around and not actually cleaning.−degamad1yObligxkcd: https://xkcd.com/1838/−fertrevinoOP1yloved the cartoon :)−oezi1yThere is some speculation that GPT-5 uses a router to decide which expert model to deploy (e.g. to mini vs o/thinking models). So the router might decide that the query can be solved by a cheaper model and this model gives worse results.
Comments
I wonder what changed with the models that created regression?
Not sure but with each release it feels like they’re just wiping the dirt around and not actually cleaning.
Obligxkcd: https://xkcd.com/1838/
loved the cartoon :)
There is some speculation that GPT-5 uses a router to decide which expert model to deploy (e.g. to mini vs o/thinking models). So the router might decide that the query can be solved by a cheaper model and this model gives worse results.