I think almost all the free LLMs (not AI) that you find on hf can 'run on CPUs'.
The claim here seems to be that it runs usefully fast on CPU.
We're not sure how accurate this claim is, because we don't know how fast this model runs on a GPU, because:
> Absent from the list of supported chips are GPUs [...]
And TFA doesn't really quantify anything, just offers:
> Perhaps more impressively, BitNet b1.58 2B4T is speedier than other models of its size — in some cases, twice the speed — while using a fraction of the memory.
The model they link to is just over 1GB in size, and there's plenty of existing 1-2GB models that are quite serviceable on even a mildly-modern CPU-only rig.
If you click the demo link, you can type a live prompt and see it run on CPU or GPU (A100). From my test, the CPU was laughably slower. To my eyes, it seems comparable to the models I can run with llama.cpp today. Perhaps I am completely missing the point of this.
Comments
I think almost all the free LLMs (not AI) that you find on hf can 'run on CPUs'.
The claim here seems to be that it runs usefully fast on CPU.
We're not sure how accurate this claim is, because we don't know how fast this model runs on a GPU, because:
And TFA doesn't really quantify anything, just offers: The model they link to is just over 1GB in size, and there's plenty of existing 1-2GB models that are quite serviceable on even a mildly-modern CPU-only rig.If you click the demo link, you can type a live prompt and see it run on CPU or GPU (A100). From my test, the CPU was laughably slower. To my eyes, it seems comparable to the models I can run with llama.cpp today. Perhaps I am completely missing the point of this.