I've had my 5080 running 24/7 over the past 2 weeks to try to understand the impact of quants on models. Came to some strange conclusions which were interesting to me.
I didn't really see a decline until going under Q4 for the most part, and MoEs were impacted far less then dense models.
Kind of cool results, and unexpected. Although this is just an initial set of tests to validate if the quant itself damaged the model, I've got a longer article coming in the future that's going to test these models in devops, coding, and long sessions. Also, I'm aware it's only a 16GiB card! I've got another article series coming out in the near future for much larger VRAM!
The benchmark code, dataset, and results are open sourced. Take a look, tell me I'm wrong (Wouldn't be the first time!), or run the benchmarks yourself!
One thing I like about this result is that it separates “the variable we expect to matter” from “the variable that actually survives measurement.”
There’s a parallel in quantitative trading. It’s very easy to build a hierarchy from theory — more precision, more parameters, a better Sharpe, a more sophisticated model — and then unconsciously treat that hierarchy as an empirical fact.
The interesting question is usually: under the conditions we actually care about, can we reliably distinguish A from B?
I’d be interested to see whether the separation between the quants becomes clearer as session length and task complexity increase. That feels like the equivalent of moving a trading model from a clean backtest into progressively less forgiving production conditions (which is kinda what I'm obsessed with).
There was a good data point on r/LLMDevs this week: two weeks of head-to-head quant testing on a 5080 showed that above Q4 most quants are statistically indistinguishable. That matches my experience — the cliff is below Q4, and it's not uniform across architectures. With MoE models the hidden cost of aggressive quants is routing drift: the router's logits get noisier, experts get picked wrong, and you lose quality in ways that don't show up on standard perplexity evals. MTP draft models are even more sensitive — a low-quant draft quietly caps your speculative-decoding acceptance rate while the headline tok/s still looks fine. Have you compared router decision distributions between Q4 and Q2, or just end-task accuracy?
Comments
Hey HN!
I've had my 5080 running 24/7 over the past 2 weeks to try to understand the impact of quants on models. Came to some strange conclusions which were interesting to me.
I didn't really see a decline until going under Q4 for the most part, and MoEs were impacted far less then dense models.
Kind of cool results, and unexpected. Although this is just an initial set of tests to validate if the quant itself damaged the model, I've got a longer article coming in the future that's going to test these models in devops, coding, and long sessions. Also, I'm aware it's only a 16GiB card! I've got another article series coming out in the near future for much larger VRAM!
The benchmark code, dataset, and results are open sourced. Take a look, tell me I'm wrong (Wouldn't be the first time!), or run the benchmarks yourself!
One thing I like about this result is that it separates “the variable we expect to matter” from “the variable that actually survives measurement.”
There’s a parallel in quantitative trading. It’s very easy to build a hierarchy from theory — more precision, more parameters, a better Sharpe, a more sophisticated model — and then unconsciously treat that hierarchy as an empirical fact.
The interesting question is usually: under the conditions we actually care about, can we reliably distinguish A from B?
I’d be interested to see whether the separation between the quants becomes clearer as session length and task complexity increase. That feels like the equivalent of moving a trading model from a clean backtest into progressively less forgiving production conditions (which is kinda what I'm obsessed with).
There was a good data point on r/LLMDevs this week: two weeks of head-to-head quant testing on a 5080 showed that above Q4 most quants are statistically indistinguishable. That matches my experience — the cliff is below Q4, and it's not uniform across architectures. With MoE models the hidden cost of aggressive quants is routing drift: the router's logits get noisier, experts get picked wrong, and you lose quality in ways that don't show up on standard perplexity evals. MTP draft models are even more sensitive — a low-quant draft quietly caps your speculative-decoding acceptance rate while the headline tok/s still looks fine. Have you compared router decision distributions between Q4 and Q2, or just end-task accuracy?