Congrats on the launch! I'll give it a try. I've been using vLLM on Linux so far but have wanted to be able to use a ggml backend - have you done any perf comparisons?
vLLM has far more throughput with batching, but that is also a WIP feature for llama.cpp.
Current standing is something like:
- vLLM is the fastest overall with batching, and has decent (but not SOTA) 4 bit quantization.
- Llama.cpp has the best hybrid CPU/GPU inference by far, has the most bells and whistles, has good and very flexible quantization, and is reasonably fast in CUDA without batching (but is getting batching soon). It has opencl and rocm backends, but support is focused on CUDA/Metal/CPU. Its the best backend for dGPUs that wont fit the whole model, and is otherwise a jack of all trades.
- MLC-LLM (with the TVM Vulkan backend) is the king of speed on IGPs, mobile devices and AMD/Intel dGPUs without having to fuss with a ROCM install. Its extremely fast on Nvidia dGPUS even without CUDA. It theoretically has "easy" support for webGPU and exotic hardware like FPGAs or AI blocks. But its 4-bit quantization was not as good as llama.cpp, last I checked.
- exLLAMAv2 has, by far, the best quantization for squeezing models onto small GPUs, and is the fastest CUDA (and ROCM?) backend with no batching. Its feature rich with a frontend like text-gen-ui
- Plain HF Transformers is... a fine default, but the master of none. The best use case is probably for testing research implementations.
He has benchmarks on an A6000 which should be roughly in line w/ a 3090 if you want to compare to my numbers (I test mlc as well, although my 3090 results are slower since I'm testing a llama2-7b @ 4K context and mlc currently slows down significantly w/ longer context): https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...
One nice thing about Ollama vs. stock llama.cpp is Ollama supports both ggml and gguf models. If you've still got a lot of old ggml bins around you can easily create a model file and use them.
I haven't done benchmarking vs. vLLM, but it's quite fast; in my tests on an A100-80g w/ llama2 70b I was getting over 25 tok/sec which is just mind blowing.
I was even getting around 30 tok/sec on llama2 7b on an old RTX 1070, which is equally crazy.
Comments
Congrats on the launch! I'll give it a try. I've been using vLLM on Linux so far but have wanted to be able to use a ggml backend - have you done any perf comparisons?
vLLM has far more throughput with batching, but that is also a WIP feature for llama.cpp.
Current standing is something like:
- vLLM is the fastest overall with batching, and has decent (but not SOTA) 4 bit quantization.
- Llama.cpp has the best hybrid CPU/GPU inference by far, has the most bells and whistles, has good and very flexible quantization, and is reasonably fast in CUDA without batching (but is getting batching soon). It has opencl and rocm backends, but support is focused on CUDA/Metal/CPU. Its the best backend for dGPUs that wont fit the whole model, and is otherwise a jack of all trades.
- MLC-LLM (with the TVM Vulkan backend) is the king of speed on IGPs, mobile devices and AMD/Intel dGPUs without having to fuss with a ROCM install. Its extremely fast on Nvidia dGPUS even without CUDA. It theoretically has "easy" support for webGPU and exotic hardware like FPGAs or AI blocks. But its 4-bit quantization was not as good as llama.cpp, last I checked.
- exLLAMAv2 has, by far, the best quantization for squeezing models onto small GPUs, and is the fastest CUDA (and ROCM?) backend with no batching. Its feature rich with a frontend like text-gen-ui
- Plain HF Transformers is... a fine default, but the master of none. The best use case is probably for testing research implementations.
Hamel Husain hasn't done testing vs llama.cpp, but this still might be of interest (includes mlc which is roughly in line w/ llama.cpp batch=1 perf): https://hamel.dev/notes/llm/inference/03_inference.html
He has benchmarks on an A6000 which should be roughly in line w/ a 3090 if you want to compare to my numbers (I test mlc as well, although my 3090 results are slower since I'm testing a llama2-7b @ 4K context and mlc currently slows down significantly w/ longer context): https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...
One nice thing about Ollama vs. stock llama.cpp is Ollama supports both ggml and gguf models. If you've still got a lot of old ggml bins around you can easily create a model file and use them.
I haven't done benchmarking vs. vLLM, but it's quite fast; in my tests on an A100-80g w/ llama2 70b I was getting over 25 tok/sec which is just mind blowing. I was even getting around 30 tok/sec on llama2 7b on an old RTX 1070, which is equally crazy.