Comment on Smaller, faster, safer: running Kimi and GLM at scaleComments−om81moWhy int4? There are a lot of superior 4 bit formats like nf4 from bitsandbytes.−Tepix1moWell, INT4 is good enough apparently and it's very fast.
Comments
Why int4? There are a lot of superior 4 bit formats like nf4 from bitsandbytes.
Well, INT4 is good enough apparently and it's very fast.