GGUF is unsupported by ComfyUI’s memory management system that enables running models much larger than fit in VRAM with tolerable efficiency via weight streaming, but for unified memory systems that system is less relevant (unless using models too big to run in unified memory AND having fast enough mass storage to benefit from direct-from-disk weight streaming.)
So I did test this, and it doesn't work because the quantized layers need torch._int_mm, which PyTorch's MPS backend doesn't implement. It just throws NotImplementedError.
Comments
GGUF is outdated in the latest versions of Comfy-UI. If you want a good balance of size, speed and quality you should use the int8_convrot model from the official Comfy Org Repo https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffus...
GGUF is unsupported by ComfyUI’s memory management system that enables running models much larger than fit in VRAM with tolerable efficiency via weight streaming, but for unified memory systems that system is less relevant (unless using models too big to run in unified memory AND having fast enough mass storage to benefit from direct-from-disk weight streaming.)
This is good advice if you have nvidia, but for Mac does not apply currently.
So I did test this, and it doesn't work because the quantized layers need torch._int_mm, which PyTorch's MPS backend doesn't implement. It just throws NotImplementedError.