1) Many models are now competitive at the top tier, including open source.
2) GLM 5.2 in particular was a major step forward in open source coding agent performance,
3) Harnesses make a huge difference in cost-performance.
4) Cheaper per-token does not imply cheaper per-task.
It takes time and effort to build such benchmark. It works at Databricks scale, I'm not sure smaller companies are ready to invest on internal benchmarks.
But they are more vendor neutral, now they don't sell their own model. It's interesting from a benchmark point of view.
Comments
1) Many models are now competitive at the top tier, including open source. 2) GLM 5.2 in particular was a major step forward in open source coding agent performance, 3) Harnesses make a huge difference in cost-performance. 4) Cheaper per-token does not imply cheaper per-task.
Also they suggest every company should build their own benchmark and repeat these tests with new models instead of relying on the SWE bench.
It takes time and effort to build such benchmark. It works at Databricks scale, I'm not sure smaller companies are ready to invest on internal benchmarks.
But they are more vendor neutral, now they don't sell their own model. It's interesting from a benchmark point of view.