Skip to content

Comment on Benchmarking coding agents on Databricks' multi-million line codebase

Comments

How is Pi so efficient? You'd think agent harness made by model makers would perform better.

Different incentives. Claude Code makes more money for Anthropic by generating larger contexts. Anthropic also recently changed their tokenizer so the exact same code input creates 30% more tokens, so there's a pattern there.

Claude code's system prompt is filled with irrelevant stuff about how CC works, so that the agent can help the user set it up. And there's no way to disable all the extra stuff, AFAIK.

There's https://github.com/skrabe/lobotomized-claude-code , which strips many of those, but I'm not sure if it is "legal" to use.

I think in general the model makers and to some extent their clients want a slightly higher pass rate over efficiency. This makes sense: for critical first week impressions clients notice pass rate much more, and only later start to grapple with cost. For example this is why High is the default reasoning for Fable, not Medium, and that choice of priorities propagates throughout the stack.

they spent very little time validating what they're doing, and it works by not doing much of anything. If you spent a month figuring out a specific model+harness, you'd be way more efficient.

Other hanresses are doing overkill so they can work with any model.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.