Comment on Ollama for Linux – Run LLMs on Linux with GPU AccelerationparentComments−brucemacd2yIt could be possible to take a similar installer approach in the future for cuBLAS or hipBLAS for AMD GPUs, so there is hope.−brucethemoose22yKoboldcpp does this: https://github.com/LostRuins/koboldcpp/releases/tag/v1.44.2They basically just ship executables for different llama.cpp backends and select the correct one with a python script, which is fine, as the executables are really small.
Comments
It could be possible to take a similar installer approach in the future for cuBLAS or hipBLAS for AMD GPUs, so there is hope.
Koboldcpp does this: https://github.com/LostRuins/koboldcpp/releases/tag/v1.44.2
They basically just ship executables for different llama.cpp backends and select the correct one with a python script, which is fine, as the executables are really small.