Comment on MLC-LLM: GPT/Llama on consumer-class GPUs and phonesparentComments−azeirah3yLlama.cpp recently added partial GPU acceleration. Model dequantization as well as some BLAS operations have been moved to GPU.It runs a lot faster if you compile with cuBLAS (nvidia) or clblast (other). GPU vram doesn't matter much since it doesn't offload the model to vram.
Comments
Llama.cpp recently added partial GPU acceleration. Model dequantization as well as some BLAS operations have been moved to GPU.
It runs a lot faster if you compile with cuBLAS (nvidia) or clblast (other). GPU vram doesn't matter much since it doesn't offload the model to vram.