Comment on MLC-LLM: GPT/Llama on consumer-class GPUs and phonesComments−cpullm3yA GPU-less machine?I've rented a server but it has no GPU. Does MLC work well through only CPU inference?I'd like to get it set-up with langchain if it does work well−junrushao1994OP3yTVM Unity, the compiler used by MLC-LLM, does support CPU and SIMD instructions on each CPU backend via LLVM, but we haven't tried it out yet. I believe llama.cpp is the best option out of box at the moment.
Comments
A GPU-less machine?
I've rented a server but it has no GPU. Does MLC work well through only CPU inference?
I'd like to get it set-up with langchain if it does work well
TVM Unity, the compiler used by MLC-LLM, does support CPU and SIMD instructions on each CPU backend via LLVM, but we haven't tried it out yet. I believe llama.cpp is the best option out of box at the moment.