Skip to content

Comment on MLC-LLM: GPT/Llama on consumer-class GPUs and phonesparent

Comments

The 7b model is ca. 6gb of VRAM so yes, it is already 4bit quantized.

There are already efforts underway with GPTQ libraries but I have found they incur a substantial performance penalty, with the benefit of consuming much lower VRAM.

EDIT: I had a look at the repo, it appears the Vicuna model is using 3bit quantization.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.