Give Me FP32 or Give Me Death?arxiv.org 3 pointsspindump89301 year ago2 commentsSaveHideCopy link On HNComments−spindump8930OP1yPeople often don't understand why LLMs can be non deterministic even with deterministic seeding, temperature, sampling. This paper shows how bad it can be with different hardware and gpu hosts.−incomingpain1yQ4 is plenty for me, I dont have the budget for FP32 lol.If money wasnt a thing, id probably not be going above Q8.
Comments
People often don't understand why LLMs can be non deterministic even with deterministic seeding, temperature, sampling. This paper shows how bad it can be with different hardware and gpu hosts.
Q4 is plenty for me, I dont have the budget for FP32 lol.
If money wasnt a thing, id probably not be going above Q8.