Mixture of experts: Make each model world-class within a single domain.
If adding one more common-sense QnA makes the calculus-bot even slightly worse at caculus, don't do it.
The “mixture of experts” concept in LLMs is a way of training a single model, it’s not based on training many different models (although that was the idea when the term was originally coined).
Comments
Mixture of experts: Make each model world-class within a single domain. If adding one more common-sense QnA makes the calculus-bot even slightly worse at caculus, don't do it.
https://en.wikipedia.org/wiki/Mixture_of_experts
The “mixture of experts” concept in LLMs is a way of training a single model, it’s not based on training many different models (although that was the idea when the term was originally coined).