Llamafile is great and love it. I run all my models using it and it’s super portable, I have tested it on windows and linux, on a powerful PC and SBC. It worked great without too my issues.
It takes about a month for the features from llama.cpp to trickle in. Also figuring the best mix of context length size to vram size to desired speed takes a while before it gets intuitive.
Comments
I thought llamafile was supposed to be the solution to "too hard to work with"?
https://github.com/Mozilla-Ocho/llamafile
Llamafile is great and love it. I run all my models using it and it’s super portable, I have tested it on windows and linux, on a powerful PC and SBC. It worked great without too my issues.
It takes about a month for the features from llama.cpp to trickle in. Also figuring the best mix of context length size to vram size to desired speed takes a while before it gets intuitive.
I thought it's "docker model" (and OCI artifacts).
llamafile is a multiplatform executable that wraps the model and a slightly modified version of llama.cpp. IIRC funded by Moz.