The 7b model is ca. 6gb of VRAM so yes, it is already 4bit quantized.
There are already efforts underway with GPTQ libraries but I have found they incur a substantial performance penalty, with the benefit of consuming much lower VRAM.
EDIT: I had a look at the repo, it appears the Vicuna model is using 3bit quantization.
Comments
The 7b model is ca. 6gb of VRAM so yes, it is already 4bit quantized.
There are already efforts underway with GPTQ libraries but I have found they incur a substantial performance penalty, with the benefit of consuming much lower VRAM.
EDIT: I had a look at the repo, it appears the Vicuna model is using 3bit quantization.