Comment on Choosing an AI model: one prompt, 11 models, different resultsparentComments−andai29dThat's probably because the actual coding benchmarks were saturated several years ago.−pistoriusp29dWhich is why you should perform your own benchmarks against your own software stack.−andai29dAgree. To this I would add, many things are saturated even for smaller models, which tend to be much cheaper and faster.On many of my tests, there was no difference in the result between the smaller and bigger model, but there was a big difference in speed and price.
Comments
That's probably because the actual coding benchmarks were saturated several years ago.
Which is why you should perform your own benchmarks against your own software stack.
Agree. To this I would add, many things are saturated even for smaller models, which tend to be much cheaper and faster.
On many of my tests, there was no difference in the result between the smaller and bigger model, but there was a big difference in speed and price.