exLLAMAv2 and GPTQ are different implementations, and currently exLLAMA is definitely the best for discrete Nvidia (and AMD?) GPUs. Its faster and uses less VRAM for the same perplexity than pretty much anything else.
But the EX2 quantization is very new, and you will have to quantize many models yourself.
But its missing some killer features of llama.cpp, like grammar based sampling.
Comments
exLLAMAv2 and GPTQ are different implementations, and currently exLLAMA is definitely the best for discrete Nvidia (and AMD?) GPUs. Its faster and uses less VRAM for the same perplexity than pretty much anything else.
But the EX2 quantization is very new, and you will have to quantize many models yourself.
But its missing some killer features of llama.cpp, like grammar based sampling.