The TL;DR of the fragment at around 6:57 in the video, for those who don't want to watch, is that Boris recommends to delete all the customizations, CLAUDE.md, skills, and other stuff you had for older models, and try using the new models without all that, because the new models can surprise you.
Steering/corrective instructions are now supposed to go into memories - which are non-portable? How do we manage these documents for team members without everyone repeating themselves? And if you remove all these documents, how do you work with other models which might use them (nevermind that claude ignored AGENTS.md for the longest time).
I am on board with not putting stuff like "write clean code" into an agent file, or using plugins for tools that are now built into the harness. I don't see enough evidence to support models being significantly better at figuring out intent, or getting the assumption correct. I've always gotten better results (as ever) with very constrained instructions, vs "fix the install".
Steering/corrective instructions are now supposed to go into memories - which are non-portable?
I’m sure this just intended to steer you to vendor lock-in. Remember, these are the same people who are so insecure/petty about their product that they won’t make it recognize the .agents/AGENTS.md standard.
Even for older models I was (and am) of the opinion that a lot of the context-overstuffing cruft people have been wrapping around their LLM usage is more of an RNG-impacting (but not always for the better) lucky charm rather than it being universally helpful.
And a lot of people latched on to it as a form of self-soothing.
"I may not write much code anymore, but I can still be an expert prompt engineer!"
Not really. The big change in newer models is the amount of RL relative to pretraining. RL makes the models stable by collapsing distributions around desired outcomes. The old tweaks like giving the agent a role ("You are an expert XYZ engineer") or stuffing the context with engineering idioms have effectively been RL'd into negligence, but they still waste tokens.
You can see this visually in older image generation models. Stable Diffusion 1.5 would produce wildly different images based on slight variations in prompt and seed, but the latest image gen models are nearly seed indifferent and can tolerate a decent amount of prompt tweaking while staying "consistent"
Yes, they keep surprising me that they still keep doing all of this: https://news.ycombinator.com/item?id=48962703 with no improvement despite all the marketing assurances that "hey you don't need to read code anymore"
Yeah, I'm going over a bunch of components a frontier model generated and while they "work" the code is quite shocking and will create significant maintenance burden. Not to mention they don't use any of our shared utilities and duplicate so much code.
I do have to still remind codex to fetch large amounts of db records in batches otherwise it will always make a first false step of a naive select all style approach. Context wise naive can be ok for small datasets though so codex didn't know the naive approach was not viable. Context management the new code plumbing these days
Context management doesn't help. I had this requirement explicitly in my AGENTS.md file. Just a minute or so into a coding session it (and Claude) would ignore/forget half of AGENTS.md (because the advertised 1M token context is nothing but marketing bullshit)
The ability to start a Claude Code session in a temporary "config free" mode, similar to how incognito works for chrome, would be a nice feature to make this easy to try out, especially since model releases are so frequent.
Yep! Only (potential) downside to --bare is it runs in print mode; can't kick off an interactive session with it. The flags I listed above are a close approximation.
Anthropic had something about API-only in the description a couple months back, but they've been waking all those back for months since the fable access rollback fallout
To be fair prompts is just English language intent abstracted for code. It is still am very much a thing. Why do you feel that harness customization is fleeting, enterprises care about them a lot for many reasons from compliance to security.
From the point of view of a developer looking to differentiate themselves. The instinct is to look for generalizations, abstractions, composability, but as Boris says, these just get subsumed into the latest model RL. So what expertise is left for the developer to invest in?
Salesman "recommends" you to continue wasting more tokens and on top of that, he "recommends" you to use only recursive loops with the latest and greatest models.
Finally he also "recommends" that you do not look at the code, or even understand it.
His "recommendations" are designed to get you to spend even more tokens and get you hooked on the Opus / Fable slot machine in order to extract as much money as possible from your wallets.
I'm waiting for the next headline that's about a codebase that existed for 10+ years and then the agents came.
Now, the codebase is managed by agents and, while the company has the source, it's as if they bought it from a vendor and pay the vendor for changes. The vendor of their own system is the AI company selling them tokens.
To be fair I think Cherny is high on his own supply, not cynically trying to sell us something. He is absolutely stoked that Claude Code is an overengineered piece of junk he doesn't understand.
Comments
The TL;DR of the fragment at around 6:57 in the video, for those who don't want to watch, is that Boris recommends to delete all the customizations, CLAUDE.md, skills, and other stuff you had for older models, and try using the new models without all that, because the new models can surprise you.
All of these harnesses should support pinning config files and tools to specific models or families of models.
It's really tiring to have to tweak everything with each model release and then watch those changes mess up cheaper models in the process.
Yeah or config older than x days...
Steering/corrective instructions are now supposed to go into memories - which are non-portable? How do we manage these documents for team members without everyone repeating themselves? And if you remove all these documents, how do you work with other models which might use them (nevermind that claude ignored AGENTS.md for the longest time).
I am on board with not putting stuff like "write clean code" into an agent file, or using plugins for tools that are now built into the harness. I don't see enough evidence to support models being significantly better at figuring out intent, or getting the assumption correct. I've always gotten better results (as ever) with very constrained instructions, vs "fix the install".
I’m sure this just intended to steer you to vendor lock-in. Remember, these are the same people who are so insecure/petty about their product that they won’t make it recognize the .agents/AGENTS.md standard.
Even for older models I was (and am) of the opinion that a lot of the context-overstuffing cruft people have been wrapping around their LLM usage is more of an RNG-impacting (but not always for the better) lucky charm rather than it being universally helpful.
And a lot of people latched on to it as a form of self-soothing.
"I may not write much code anymore, but I can still be an expert prompt engineer!"
I wrote this last year, it's still true:
--- start quote ---
https://dmitriid.com/prompting-llms-is-not-engineering
In reality these are just shamanic rituals with outcomes based on faith, fear, or excitement. Engineering it is not.
--- end quote ---
There are some prompts useful for the user like brainstorming [1] but on the whole it's nothing but lucky charms
[1] brainstorming from superpowers: https://github.com/obra/superpowers
Not really. The big change in newer models is the amount of RL relative to pretraining. RL makes the models stable by collapsing distributions around desired outcomes. The old tweaks like giving the agent a role ("You are an expert XYZ engineer") or stuffing the context with engineering idioms have effectively been RL'd into negligence, but they still waste tokens.
You can see this visually in older image generation models. Stable Diffusion 1.5 would produce wildly different images based on slight variations in prompt and seed, but the latest image gen models are nearly seed indifferent and can tolerate a decent amount of prompt tweaking while staying "consistent"
Yes, they keep surprising me that they still keep doing all of this: https://news.ycombinator.com/item?id=48962703 with no improvement despite all the marketing assurances that "hey you don't need to read code anymore"
Yeah, I'm going over a bunch of components a frontier model generated and while they "work" the code is quite shocking and will create significant maintenance burden. Not to mention they don't use any of our shared utilities and duplicate so much code.
I do have to still remind codex to fetch large amounts of db records in batches otherwise it will always make a first false step of a naive select all style approach. Context wise naive can be ok for small datasets though so codex didn't know the naive approach was not viable. Context management the new code plumbing these days
Context management doesn't help. I had this requirement explicitly in my AGENTS.md file. Just a minute or so into a coding session it (and Claude) would ignore/forget half of AGENTS.md (because the advertised 1M token context is nothing but marketing bullshit)
The ability to start a Claude Code session in a temporary "config free" mode, similar to how incognito works for chrome, would be a nice feature to make this easy to try out, especially since model releases are so frequent.
You can do this but you have to pass a million long options like --disable-system-prompt --allowed-tools "" etc
I have some wrappers but the only one I really use is a variant of that pinned to haiku for quick questions.
`claude --bare --system-prompt ""` is a good starting point!
Yep! Only (potential) downside to --bare is it runs in print mode; can't kick off an interactive session with it. The flags I listed above are a close approximation.
Anthropic had something about API-only in the description a couple months back, but they've been waking all those back for months since the fable access rollback fallout
There’s really not much to specialize in and differentiate yourself. All this harness customization is fleeting. Remember prompt engineering?
To be fair prompts is just English language intent abstracted for code. It is still am very much a thing. Why do you feel that harness customization is fleeting, enterprises care about them a lot for many reasons from compliance to security.
From the point of view of a developer looking to differentiate themselves. The instinct is to look for generalizations, abstractions, composability, but as Boris says, these just get subsumed into the latest model RL. So what expertise is left for the developer to invest in?
Salesman "recommends" you to continue wasting more tokens and on top of that, he "recommends" you to use only recursive loops with the latest and greatest models.
Finally he also "recommends" that you do not look at the code, or even understand it.
His "recommendations" are designed to get you to spend even more tokens and get you hooked on the Opus / Fable slot machine in order to extract as much money as possible from your wallets.
News at 10.
I'm waiting for the next headline that's about a codebase that existed for 10+ years and then the agents came.
Now, the codebase is managed by agents and, while the company has the source, it's as if they bought it from a vendor and pay the vendor for changes. The vendor of their own system is the AI company selling them tokens.
To be fair I think Cherny is high on his own supply, not cynically trying to sell us something. He is absolutely stoked that Claude Code is an overengineered piece of junk he doesn't understand.
Boris Cherny is most definitely the kind of guy who sniffs his own farts.
He can be informative to listen to, as long as you keep that in mind.
That is the interesting part to me. Also, was anyone else familiar with the LLM-app terms "ablation" and "product overhang," prior to this? I was not.
If you were, please share the knowledge with us. Where can I learn more?
Ablation is a medical term, he's just using it generically to describe removing existing config.
I only knew ablation from spaceflight re-entry. I should have been more clear. I had never heard those words in terms of software products.
CHANGELOG: surprise me
What if I don't use customizations at all, bare prompt per feature or function only? Do those customizations help?
Yup. That's pretty much the most interesting part.
I was using Opus 4.6 until 2 days ago with no CLAUDE.md or anything and it was great.
Tried out Opus 5 and it's been a super annoying experience out of the box.