Comment on Ternary Bonsai: Top Intelligence at 1.58 BitsparentComments−haellsigh4moFyi, I believe `--flash-attn on` doesn't do anything, you should instead use `--flash-attn 1`. I'm getting ~150t/s on a RTX 3080 10GB as well with f16 cache type.−freakynit4moThanks.. updated my local docs :)
Comments
Fyi, I believe `--flash-attn on` doesn't do anything, you should instead use `--flash-attn 1`. I'm getting ~150t/s on a RTX 3080 10GB as well with f16 cache type.
Thanks.. updated my local docs :)