Very cool. Does anyone know exactly how out of luck us AMD folk are? I know there are efforts out there, but I'm kind of hoping for something "as easy as this?"
We don't current compile in CLBlast or ROCm support but if there's a lot of demand for this, we'll definitely add it in the future. One concern is not wanting to bloat out the binary size too much (CUDA is already huge!) but given how big the LLM models are anyway, maybe it's not a huge concern.
Possibly, but a core guiding principle for us is to keep everything as simple as possible. We're a small project, so if we add too many features/permutations it really makes it hard to keep everything working!
I don't believe that AMD/NVIDIA is a low entropy bit so far as configuration permutations go. Although NVIDIA is far more widespread, AMD has significant market share. The Darwin bit you already facilitate for is probably lower entropy.
They basically just ship executables for different llama.cpp backends and select the correct one with a python script, which is fine, as the executables are really small.
llama.cpp added ROCm linux support a little over a month ago. Details on that can be found in their readme: https://github.com/ggerganov/llama.cpp#hipblas (no luck for windows users though)
Comments
Very cool. Does anyone know exactly how out of luck us AMD folk are? I know there are efforts out there, but I'm kind of hoping for something "as easy as this?"
We don't current compile in CLBlast or ROCm support but if there's a lot of demand for this, we'll definitely add it in the future. One concern is not wanting to bloat out the binary size too much (CUDA is already huge!) but given how big the LLM models are anyway, maybe it's not a huge concern.
AMD support would be amazing <3
I get the boot concern, and the maintenance concern (!!!), but as you say, these models are already quite huge anyway :)
Offer two builds :) One AMD and one NVIDIA.
Possibly, but a core guiding principle for us is to keep everything as simple as possible. We're a small project, so if we add too many features/permutations it really makes it hard to keep everything working!
I don't believe that AMD/NVIDIA is a low entropy bit so far as configuration permutations go. Although NVIDIA is far more widespread, AMD has significant market share. The Darwin bit you already facilitate for is probably lower entropy.
Also, some of us weird folk have both an AMD and NVIDIA GPU in one machine...
Actually, I guess that is more common now that AMD CPUs have a GPU built in?
You could download both binaries for that scenario.
It could be possible to take a similar installer approach in the future for cuBLAS or hipBLAS for AMD GPUs, so there is hope.
Koboldcpp does this: https://github.com/LostRuins/koboldcpp/releases/tag/v1.44.2
They basically just ship executables for different llama.cpp backends and select the correct one with a python script, which is fine, as the executables are really small.
llama.cpp added ROCm linux support a little over a month ago. Details on that can be found in their readme: https://github.com/ggerganov/llama.cpp#hipblas (no luck for windows users though)
This one is basically SOTA for AMD, if you can install rocm properly:
https://github.com/YellowRoseCx/koboldcpp-rocm
Some other projects support rocm less explicitly, and not as easily.