Comment on Microsoft researchers developed a hyper-efficient AI model that can run on CPUsparentComments−falcor841yThanks, but I've skimmed through both and couldn't find an answer on why they call it "1-bit".−AzN1337c0d3r1yThe original BitNet paper (https://arxiv.org/pdf/2310.11453) BitNet: Scaling 1-bit Transformers for Large Language Models was actually binary (weights of -1 or 1),but then in the follow-up paper they started using 1.58bit weights (https://arxiv.org/pdf/2402.17764) The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits This seems to be first source of the confounding of "1-bit LLM" and ternary weights that I could find. In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}.−LeonB1yIt’s “1-bit, for particularly large values of ‘bit’”−biomcgary1yShould be 1-trit.
Comments
Thanks, but I've skimmed through both and couldn't find an answer on why they call it "1-bit".
The original BitNet paper (https://arxiv.org/pdf/2310.11453)
was actually binary (weights of -1 or 1),but then in the follow-up paper they started using 1.58bit weights (https://arxiv.org/pdf/2402.17764)
This seems to be first source of the confounding of "1-bit LLM" and ternary weights that I could find.It’s “1-bit, for particularly large values of ‘bit’”
Should be 1-trit.