load_in_4bit requires storing a fully unquantized model as well as having enough RAM to load the unquantized model.
If you're an enthusiast with 10 models downloaded, do you want that taking up 500GB or 150GB? Do you want to need 64GB of RAM to load a model, or just 16GB?
That's the main reason for the popularity of pre-quantization.
Does it? I loaded Guanaco-65B on a 48GB graphics card using the new flag and it loaded just fine. I don't think the unquantized model fits on a 48GB card. So I'm not quite sure if this is true.
I'm not sure it could be for this complex quantization scheme. It's not a simple round to nearest quantization. Even if it is possible, it would require someone to develop that functionality. Currently it does load the entire unquantized model into RAM first.
Comments
load_in_4bit requires storing a fully unquantized model as well as having enough RAM to load the unquantized model.
If you're an enthusiast with 10 models downloaded, do you want that taking up 500GB or 150GB? Do you want to need 64GB of RAM to load a model, or just 16GB?
That's the main reason for the popularity of pre-quantization.
Does it? I loaded Guanaco-65B on a 48GB graphics card using the new flag and it loaded just fine. I don't think the unquantized model fits on a 48GB card. So I'm not quite sure if this is true.
RAM, not VRAM. The model is fully loaded into (regular, system) RAM then quantized while being loaded into VRAM.
So you still need enough system RAM (or RAM+Swap) to load the unquantized model.
Why would it need to be loaded fully into system RAM at once? Couldn't it be quantized on the fly, as it's being read from disk?
Last time I looked into this, the answer was "because huggingface transformers and torch.load are written to do it this way"
You could absolutely do something streaming, or mmap the weights instead of loading them into system RAM. Just the default interfaces don't.
I'm not sure it could be for this complex quantization scheme. It's not a simple round to nearest quantization. Even if it is possible, it would require someone to develop that functionality. Currently it does load the entire unquantized model into RAM first.