Is this a best practice? It seems like we've been delivered evidence that it is really hard to do good benchmarks unless you're already intimately familiar with what you're testing, which says something about how hard it is to make a good choice.
I don't know about other industries, but this sort of result is what stuff like the STAC M3 Benchmark suite was designed for: Typical usecases that experts can implement so you can get realistic performance comparisons.
Comments
Is this a best practice? It seems like we've been delivered evidence that it is really hard to do good benchmarks unless you're already intimately familiar with what you're testing, which says something about how hard it is to make a good choice.
I don't know about other industries, but this sort of result is what stuff like the STAC M3 Benchmark suite was designed for: Typical usecases that experts can implement so you can get realistic performance comparisons.