For over a year, I've run the Little Dorrit benchmark. It's a vision task that converts handwritten editor marks on a printed page into structured JSON output.
I hadn't updated the benchmark in several months, but there are some interesting findings. Fable 5 takes the top spot (0.6579), setting a new performance record, while Kimi K3 is within a hair's breadth of its performance.
The most significant finding is that Opus 4.8 regresses drastically compared to Opus 4.7, from 0.4805 to 0.2150. This seems mainly due to a regression in its ability to count line numbers, and it's something you might want to keep in mind when designing your own agents.
Comments
For over a year, I've run the Little Dorrit benchmark. It's a vision task that converts handwritten editor marks on a printed page into structured JSON output.
I hadn't updated the benchmark in several months, but there are some interesting findings. Fable 5 takes the top spot (0.6579), setting a new performance record, while Kimi K3 is within a hair's breadth of its performance.
The most significant finding is that Opus 4.8 regresses drastically compared to Opus 4.7, from 0.4805 to 0.2150. This seems mainly due to a regression in its ability to count line numbers, and it's something you might want to keep in mind when designing your own agents.