CPU inference for LLMs takes forever (you'll get like 1tk/s on CPU) and limits you significantly in terms of model size/quality. You'll lock up all of your cores to provide service for a single user at a snail's pace.
I don't think it should even be considered as an option
llama 3.2 3b, qwen2.5 3B quantized to 4bit runs CPU inference quite fast. You can get a beefier VM and still save a ton of money. Depending on the context token length of this soluion, it's either fast or slow. If it's below 1024 tokens per request, you get around 10 sec delay, if you are at around 128 tokens I guess you would be somewhere at 1 sec for time to first token...
Comments
You could use CPU inference on a smaller local model (either always or after a demo budget is spent).
Don't forget to put up a "donate" button.
That sounds like a bad idea to me.
CPU inference for LLMs takes forever (you'll get like 1tk/s on CPU) and limits you significantly in terms of model size/quality. You'll lock up all of your cores to provide service for a single user at a snail's pace.
I don't think it should even be considered as an option
llama 3.2 3b, qwen2.5 3B quantized to 4bit runs CPU inference quite fast. You can get a beefier VM and still save a ton of money. Depending on the context token length of this soluion, it's either fast or slow. If it's below 1024 tokens per request, you get around 10 sec delay, if you are at around 128 tokens I guess you would be somewhere at 1 sec for time to first token...
My point still stands … 10 seconds is an eternity, and a 3B model isn’t that performant.
I’d rather not offer a demo at all than offer it with these parameters.