In our multi-agent game coding evaluations, we usually see Chinese models struggle in one-shot reasoning but make up for it with tool use and iterating towards better solutions. Kimi K3 follows that pattern, ranking 19th in one-shot coding and 3rd in agentic coding (where the model gets a harness and tools and many calls to iterate toward better code). Only Sol and Fable have better average agentic coding submissions.
Agentic coding is what's most relevant to software engineers, but so is speed, which is a real usability issue right now. Third-party inference providers like Fireworks have bridged the gap for some previous releases.
It's exciting to see the open weights frontier becoming the norm. Competing on having the frontier model is going to be an increasingly difficult business.
Comments
In our multi-agent game coding evaluations, we usually see Chinese models struggle in one-shot reasoning but make up for it with tool use and iterating towards better solutions. Kimi K3 follows that pattern, ranking 19th in one-shot coding and 3rd in agentic coding (where the model gets a harness and tools and many calls to iterate toward better code). Only Sol and Fable have better average agentic coding submissions.
Agentic coding is what's most relevant to software engineers, but so is speed, which is a real usability issue right now. Third-party inference providers like Fireworks have bridged the gap for some previous releases.
It's exciting to see the open weights frontier becoming the norm. Competing on having the frontier model is going to be an increasingly difficult business.
Data at https://gertlabs.com/rankings?mode=agentic_coding