Something like this is nice, where instead of having 1 model with X active experts, you have 10 different models, all small and dense, trained on specific information. and loaded on 10 different servers, with one router.
This is actually how I develop and use the mesh at home. Rather than splitting models, I aggregate disparate compute behind one endpoint, without having separate inference providers on each host and a gateway like LiteLLM
True, I've been telling that coding LLMs have been detrimental to local AI scene. What happened to those nifty ML projects which can run on raspberry pi zero.
Comments
Something like this is nice, where instead of having 1 model with X active experts, you have 10 different models, all small and dense, trained on specific information. and loaded on 10 different servers, with one router.
This is actually how I develop and use the mesh at home. Rather than splitting models, I aggregate disparate compute behind one endpoint, without having separate inference providers on each host and a gateway like LiteLLM
True, I've been telling that coding LLMs have been detrimental to local AI scene. What happened to those nifty ML projects which can run on raspberry pi zero.