Comment on GPT-4o mini: advancing cost-efficient intelligenceparentComments−pants22yNot really, MoE is trained all at once and the 'experts' don't have pre-defined specializations. They end up being more like "punctuation expert" and "pronoun expert" than "math expert" and "french expert"−laborcontract2yHaven't tried any yet, but it sounds like parent may be interested in an LLM router. https://github.com/lm-sys/RouteLLM
Comments
Not really, MoE is trained all at once and the 'experts' don't have pre-defined specializations. They end up being more like "punctuation expert" and "pronoun expert" than "math expert" and "french expert"
Haven't tried any yet, but it sounds like parent may be interested in an LLM router. https://github.com/lm-sys/RouteLLM