Comment on QLoRA: Efficient Finetuning of Quantized LLMsparentComments−ianpurton3yAssuming you have 24GB of VRAM, then you should be able to run something like 7 billion parameters i.e. MPT-7b and quantized perhaps you can get up to 13 billion.Note: I have no practical experience of this, just reading around.−MacsHeadroom3y24GB can fit 33B parameter models in 4bit. You only need 4GB to run 7B models.
Comments
Assuming you have 24GB of VRAM, then you should be able to run something like 7 billion parameters i.e. MPT-7b and quantized perhaps you can get up to 13 billion.
Note: I have no practical experience of this, just reading around.
24GB can fit 33B parameter models in 4bit. You only need 4GB to run 7B models.