Something even cooler would be a model trained for 4 or less (1.33) bit weights instead of quantized after pretraining.

Math units are completely underutilized when I'm inferencing with batch size of 1, and post-training quantization under 8 bits loses too much of the precision to make a real difference compared to smaller models with higher precision.