Parameter size and total number of parameters so ultimately the total size of the model in memory.
This leads to some interesting optimizations. You can quantize all the parameters (or certain layers) of a model and halve or quarter the memory requirement but maintain most of the model's intelligence. This increases the token rate inversely with the size reduction.
Popular quantizations for local models are 8-bit and 4-bit parameter sizes. The Blackwell series of nVidia chips now even support native FP4 math making 4-bit quantizations even faster.
Comments
Parameter size and total number of parameters so ultimately the total size of the model in memory.
This leads to some interesting optimizations. You can quantize all the parameters (or certain layers) of a model and halve or quarter the memory requirement but maintain most of the model's intelligence. This increases the token rate inversely with the size reduction.
Popular quantizations for local models are 8-bit and 4-bit parameter sizes. The Blackwell series of nVidia chips now even support native FP4 math making 4-bit quantizations even faster.