I guess we are reaching the point where “10T parsmeters” sounds more like a marketing number than a meaningful metric.
Between moE, aggressive quantization, and synthetic data pipelines, it’s getting harder to tell whether bigger models are actually better, or just more expensive to train.
Would be more interesting to see ->
capability per dollar or per watt, not parameter count...
Comments
I guess we are reaching the point where “10T parsmeters” sounds more like a marketing number than a meaningful metric.
Between moE, aggressive quantization, and synthetic data pipelines, it’s getting harder to tell whether bigger models are actually better, or just more expensive to train.
Would be more interesting to see -> capability per dollar or per watt, not parameter count...