Comment on Mesh LLM: distributed AI computing on irohComments−josefrichter2moIs there a catch? If not, this would be super useful.−stymaar2moThe catch is that the token generation speed is going to be limited by network latency, making it unbearably slow to run over the internet.It can be great on a local network though, especially if your workload is prefill-heavy (more text input to process than output tokens to emit).
Comments
Is there a catch? If not, this would be super useful.
The catch is that the token generation speed is going to be limited by network latency, making it unbearably slow to run over the internet.
It can be great on a local network though, especially if your workload is prefill-heavy (more text input to process than output tokens to emit).