Quantizing is porting the weights from using high-precision floating point decimals to lower-precision numbers, e.g., 4-bit and 8-bit ints. Less memory used by the weights means we can run bigger models on the same hardware.
The idea behind quantization is that these models have so many parameters, they'll still work even if we reduce each node's flexibility.
In the same breath, there's no free lunch. There's always a trade-off. Sure, the model might now fit in your VRAM, but it might be less accurate for your specific task.
Comments
Quantizing is porting the weights from using high-precision floating point decimals to lower-precision numbers, e.g., 4-bit and 8-bit ints. Less memory used by the weights means we can run bigger models on the same hardware.
The idea behind quantization is that these models have so many parameters, they'll still work even if we reduce each node's flexibility.
In the same breath, there's no free lunch. There's always a trade-off. Sure, the model might now fit in your VRAM, but it might be less accurate for your specific task.