It’s objectively worse in my local tests compared to Mistral. Again their model doesn’t include MT-bench benchmark because it’s really really bad at answering a follow up question(s). (this is also a problem in Ultra). It’s reasoning is also pretty bad compared to mistral.
Comments
It’s objectively worse in my local tests compared to Mistral. Again their model doesn’t include MT-bench benchmark because it’s really really bad at answering a follow up question(s). (this is also a problem in Ultra). It’s reasoning is also pretty bad compared to mistral.
I can’t get it to recognize the stop token consistently in the 7b models.
About 50% of the shots, I get a sentence and a half of beautiful poetry, then a codeswitch into kanji, and then ral ral ral ral ral 膳 ral 杯 ral ral
Until I kill the process. Not every time, but way more often than the other llamas (which is basically never, these days).
I think they underestimated the impact of training on bulleted lists. It seems to love those!