Skip to content

Comment on Benchmarking coding agents on Databricks' multi-million line codebaseparent

Comments

I think in general the model makers and to some extent their clients want a slightly higher pass rate over efficiency. This makes sense: for critical first week impressions clients notice pass rate much more, and only later start to grapple with cost. For example this is why High is the default reasoning for Fable, not Medium, and that choice of priorities propagates throughout the stack.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.