There was a good data point on r/LLMDevs this week: two weeks of head-to-head quant testing on a 5080 showed that above Q4 most quants are statistically indistinguishable. That matches my experience — the cliff is below Q4, and it's not uniform across architectures. With MoE models the hidden cost of aggressive quants is routing drift: the router's logits get noisier, experts get picked wrong, and you lose quality in ways that don't show up on standard perplexity evals. MTP draft models are even more sensitive — a low-quant draft quietly caps your speculative-decoding acceptance rate while the headline tok/s still looks fine. Have you compared router decision distributions between Q4 and Q2, or just end-task accuracy?
Comments
There was a good data point on r/LLMDevs this week: two weeks of head-to-head quant testing on a 5080 showed that above Q4 most quants are statistically indistinguishable. That matches my experience — the cliff is below Q4, and it's not uniform across architectures. With MoE models the hidden cost of aggressive quants is routing drift: the router's logits get noisier, experts get picked wrong, and you lose quality in ways that don't show up on standard perplexity evals. MTP draft models are even more sensitive — a low-quant draft quietly caps your speculative-decoding acceptance rate while the headline tok/s still looks fine. Have you compared router decision distributions between Q4 and Q2, or just end-task accuracy?