The rate at which the Transformer architecture can learn and adapt to new information is astounding - so it makes absolutely no sense to me that they can't manage to throw 200 data annotators at their model and make it best-in-class.
They did sink capital into Anthropic recently so perhaps that shuffled things up internally / changed some roadmaps?
If that's the case - Anthropic's models are extremely capable and can be easily launched to the top of the leaderboards if they go for a less belligerent training approach.
Comments
The rate at which the Transformer architecture can learn and adapt to new information is astounding - so it makes absolutely no sense to me that they can't manage to throw 200 data annotators at their model and make it best-in-class.
They did sink capital into Anthropic recently so perhaps that shuffled things up internally / changed some roadmaps?
If that's the case - Anthropic's models are extremely capable and can be easily launched to the top of the leaderboards if they go for a less belligerent training approach.
these foundation models are trained on trillions of tokens to achieve quality. 200 data annotators with unknown quality issues may be not enough..
Not to mention that you don’t think OpenAI is paying probably many, many more annotators?
Not to mention the free feedback they get from the 100 million chatgpt users.
google has their own piles of data