I’m not an expert, but my understanding is that MTPs are smaller LLMs fine-tuned to "mimic" / predict a specific model’s response. It’s possible that the MTP you’re using isn’t trained well enough on Qwen 3.8. What accept rate are you getting?
Acceptance rate is good, but MTP doesn't help in my case because of my machine's memory bandwidth constraints (M4 Max). Turn out it's better to turn MTP off.
Comments
I’m not an expert, but my understanding is that MTPs are smaller LLMs fine-tuned to "mimic" / predict a specific model’s response. It’s possible that the MTP you’re using isn’t trained well enough on Qwen 3.8. What accept rate are you getting?
Acceptance rate is good, but MTP doesn't help in my case because of my machine's memory bandwidth constraints (M4 Max). Turn out it's better to turn MTP off.
For qwen, it's an additional transformer layer at the very back, it ships as part of the model.