lhl on Ornith-1.0: self-improving open-source models for agentic coding

Comment on Ornith-1.0: self-improving open-source models for agentic coding

I've been testing Ornith-1.0 35B (my own FP8-block quant) and I like it. It runs at >200 tok/s w/ vLLM on an RTX PRO 6000 (sm120), I've run >140M cached tokens of agentic coding work on it over the past few days. It seems to about somewhere between Qwen 3.6 35B-A3B and 27B, but the good thing: it overthinks/doom-loop a lot less than Qwen 3.6. When looking at the thinking traces I like its breakdown approach template.

It does good job on basic analysis, tasks, and some front-end/backend changes on a medium-sized Go codebase, but it reached its limits totally botching a longer (simple) kernel implementation job (about 100 iterations in Pi Agent harness) - this is the type of thing that stronger open models (Kimi K2.6, GLM 5.2) are able to do.

regularfry 2mo

With this model size I've found that the harness seems to matter more. I've moved on to little-coder rather than raw pi with qwen3.6 27b personally, it might be worth taking a look.

lhl 2mo

You got me curious, so I made a little harness comparison to my model test suite:

    Model                              Adapter       Suite           Score   Passed  Tasks
    ---------------------------------  ------------  --------------  ------  ------  -----
    local/ornith-1.0-35b               little_coder  aider_polyglot  36.0%   81/225  225
    local/ornith-1.0-35b               pi_devstack   aider_polyglot  39.6%   89/225  225
    local/ornith-1.0-35b               pi_vanilla    aider_polyglot  32.0%   72/225  225

Little Code does a little better than raw Pi, although maybe not better than my personal Pi setup: https://github.com/lhl/devstack

Comments