Comment on Kaist develops next-generation ultra-low power LLM acceleratorparentComments−dloss2yWhich of the model requirements of Coral TPUs [1] are the most problematic for LLMs?[1] https://coral.ai/docs/edgetpu/models-intro/#model-requiremen...−semisight2yGuessing as to what the GP meant--coral TPUs max out around 8M parameters, IIRC. That's a few orders of magnitude less than the smallest LLM model.−moffkalast2yThe part where they have like 3 bytes of memory so you switch from extremely high latency of RAM to laughably sluggish latency of USB serial. I think there's also no support below 8 bit quants, which you'd really need.
Comments
Which of the model requirements of Coral TPUs [1] are the most problematic for LLMs?
[1] https://coral.ai/docs/edgetpu/models-intro/#model-requiremen...
Guessing as to what the GP meant--coral TPUs max out around 8M parameters, IIRC. That's a few orders of magnitude less than the smallest LLM model.
The part where they have like 3 bytes of memory so you switch from extremely high latency of RAM to laughably sluggish latency of USB serial. I think there's also no support below 8 bit quants, which you'd really need.