Comment on QLoRA: Efficient Finetuning of Quantized LLMsComments−epups3yDo you know which model size can be run with a 3090?−redox993y3090 can handle the ~30B models quantized to 4 bits.−ianpurton3yAssuming you have 24GB of VRAM, then you should be able to run something like 7 billion parameters i.e. MPT-7b and quantized perhaps you can get up to 13 billion.Note: I have no practical experience of this, just reading around.−MacsHeadroom3y24GB can fit 33B parameter models in 4bit. You only need 4GB to run 7B models.
Comments
Do you know which model size can be run with a 3090?
3090 can handle the ~30B models quantized to 4 bits.
Assuming you have 24GB of VRAM, then you should be able to run something like 7 billion parameters i.e. MPT-7b and quantized perhaps you can get up to 13 billion.
Note: I have no practical experience of this, just reading around.
24GB can fit 33B parameter models in 4bit. You only need 4GB to run 7B models.