Comment on German AI consortium releases Soofi S, an open 30B model that tops benchmarksComments−jbellis1moIt tops benchmarks because it uses them in its training data. https://x.com/eliebakouch/status/2077425801633427919−krackers1motraining on the test set is all you need−ch4s31moIt's ready for law school!−fxwin1molooks like they have since fixed the dataset, hope they release an updated report (either without the leaked evals or a new checkpoint/weights)https://huggingface.co/datasets/AIML-TUDA/QA-base/commit/2f9...−sva_1mohttps://nitter.net/eliebakouch/status/2077425801633427919That is very disappointing.−Tostino1moThat's just embarrassing. It keeps happening though.−amaiOP1moIt seems they corrected this already: https://nitter.net/jphme/status/2077468512025477226#m−jbellis1moI guess you didn't read ellie's reply directly underneath that?−istvan01moReminds me of: https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal−AtlasBarfed1moscrape the scrapers... all's fair...−sajithdilshan1moDisappointing. Sounds like Volkswagen 2.0
Comments
It tops benchmarks because it uses them in its training data. https://x.com/eliebakouch/status/2077425801633427919
training on the test set is all you need
It's ready for law school!
looks like they have since fixed the dataset, hope they release an updated report (either without the leaked evals or a new checkpoint/weights)
https://huggingface.co/datasets/AIML-TUDA/QA-base/commit/2f9...
https://nitter.net/eliebakouch/status/2077425801633427919
That is very disappointing.
That's just embarrassing. It keeps happening though.
It seems they corrected this already: https://nitter.net/jphme/status/2077468512025477226#m
I guess you didn't read ellie's reply directly underneath that?
Reminds me of: https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal
scrape the scrapers... all's fair...
Disappointing. Sounds like Volkswagen 2.0