Comment on Quantization from the Ground UpparentComments−faangguyindia5moU are better off just buying their coding plan.Running LLM makes no sense whatsoever−oompydoompy745moRemaining dependent on proprietary frontier models that you can only access via an API makes no sense whatsoever. My hope is that the future is open weight models running on local hardware.−naasking5moEventually, yes. ParoQuant is hopefully the future here, 4-bit weights with no real degradation from FP16:https://github.com/z-lab/paroquant
Comments
U are better off just buying their coding plan.
Running LLM makes no sense whatsoever
Remaining dependent on proprietary frontier models that you can only access via an API makes no sense whatsoever. My hope is that the future is open weight models running on local hardware.
Eventually, yes. ParoQuant is hopefully the future here, 4-bit weights with no real degradation from FP16:
https://github.com/z-lab/paroquant