I wanted to try without lookup tables to at least have a baseline, and also because the fixed point packing idea lent itself naturally to using multiplications by powers of 3 when unpacking.
Thanks for taking the time to reply! I haven’t done any serious low level optimisation on modern CPUs so most of my intuitions are probably way out of date.
Once you know how to compress 32-bit parameters to ternary, compressing ternary to binary is the easy part. :)
They would keep re-compressing the model in its entirety, recursively until the whole thing was a single bit, but the unpacking and repacking during inference is a bitch.
1.58 is still more than 1 in general unless the parameters are corelated. At 1 bit it seems unlikely that you could pack/unpack independent parameters reliably without additional data.
Comments
Why do they call it "1-bit" if it uses ternary {-1, 0, 1}? Am I missing something?
https://compilade.net/blog/ternary-packing is a good explainer (previous discussion https://news.ycombinator.com/item?id=42329307)
Thanks, but I've skimmed through both and couldn't find an answer on why they call it "1-bit".
The original BitNet paper (https://arxiv.org/pdf/2310.11453)
was actually binary (weights of -1 or 1),but then in the follow-up paper they started using 1.58bit weights (https://arxiv.org/pdf/2402.17764)
This seems to be first source of the confounding of "1-bit LLM" and ternary weights that I could find.It’s “1-bit, for particularly large values of ‘bit’”
Should be 1-trit.
That’s pretty cool. :) One thing I don’t get is why do multiple operations when a 243-entry lookup table would be simpler and hopefully faster?
Because lookup tables are not necessarily faster compared to 8-bit SIMD operations, at least when implemented naïvely.
Lookup tables can be fast, but it's not simpler, see T-MAC https://arxiv.org/abs/2407.00088 (Note that all comparisons with `llama.cpp` were made before I introduced the types from https://github.com/ggml-org/llama.cpp/pull/8151 where the 1.6-bit type uses the techniques described in the aforementioned blog post).
I wanted to try without lookup tables to at least have a baseline, and also because the fixed point packing idea lent itself naturally to using multiplications by powers of 3 when unpacking.
Thanks for taking the time to reply! I haven’t done any serious low level optimisation on modern CPUs so most of my intuitions are probably way out of date.
Maybe they are rounding down from 1.5-bit :)
Classic Microsoft naming shenanigans.
It's not too late to claim 1bitdotnet.net before they do.
LLM Series One S and X
Once you know how to compress 32-bit parameters to ternary, compressing ternary to binary is the easy part. :)
They would keep re-compressing the model in its entirety, recursively until the whole thing was a single bit, but the unpacking and repacking during inference is a bitch.
There are about 1.58 (i.e. log_2(3)) bits per digit, so they just applied the constant function that maps the reals to 1 to it.
1.58 is still more than 1 in general unless the parameters are corelated. At 1 bit it seems unlikely that you could pack/unpack independent parameters reliably without additional data.
I like that as an explanation, but then every system is 1-bit, right? It definitely would simplify things.
see also https://arxiv.org/pdf/2310.11453