What you call "Weathering" has been a constant gripe of mine. We have LLM-driven softwares toward that almost seem to start from scratch every time a request comes in - there are mechanisms to learn or generalize, like writing out a memory, but they are not reliable or reliable in general. There is no convenient lever to be able to say "yes this is in the memory but the request seems like it needs a fresh scan of data, so ignore your memory", or the opposite "you can infer this from stuff in the memory - don't re-reason!". There is some work like Dynamic Cheatsheets [1] and Agentic Context Engineering [2] that have studied this aspect, but we are far from a generally reliable solution. And till we have that, I think the system variations for systems trying to solve this problem are going to be (a) LLM-leaning: create unstructured memory files, with human in the loop as a filter to reject inaccurate responses (b) LLM-as-a-layer: what you describe and the article kind of is doing.
I’ve been thinking about this.. the same idea behind Dynamic cheat sheets paper and the ACE paper. Thanks for those links! Nice to see those ideas academically studied.
My problem is the context of today’s models (that I’ve been using - Claude Opus 5 and GPT-Sol) are a black box to a user like me. I cannot tell what they already hold in their context over the duration of a coding session. I cannot tell influencers a little by using/not using MCP tools, AGENTS.md like MD files, but what they _actually_ use at inference time is still opaque.
For example, if I insist that GPT-Sol use a size bounded dynamic cheat sheet for my project, and even have those reflector&curator agents refine them concurrently, what is to say GPT-Sol doesn’t have an intrinsic cheat sheet like that already, and it is just reading/curating/managing this one in addition and only getting confused needlessly along the way?
Similarly, all those codegraph like tools and semantic search addons.. what is to say the model genuinely uses them instead of relying on its own inference techniques and just “perform” for the user audience as if they are using these fancy tools?
This is not to say we shouldn’t try more ways of managing context and just give up to our AI provider overlords. Just thinking through the logic of what we are working with.
> My problem is the context of today’s models (that I’ve been using - Claude Opus 5 and GPT-Sol) are a black box to a user like me.
Would it even help if it weren’t a blackbox? Maybe to the frontier labs with resources, but even trying to read through the thinking traces of an open model doing a nontrivial task easily takes me an entire day to do correctly (i.e. actually looking at the code it read in) and much more if I’m trying to tune a specific repeatable task.
Applying that to agentic coding seems like a losing proposition. The only way to manage it seems to be more agents and that’s its own rabbit hole of compounding errors (like Claude’s useless comments).
Comments
What you call "Weathering" has been a constant gripe of mine. We have LLM-driven softwares toward that almost seem to start from scratch every time a request comes in - there are mechanisms to learn or generalize, like writing out a memory, but they are not reliable or reliable in general. There is no convenient lever to be able to say "yes this is in the memory but the request seems like it needs a fresh scan of data, so ignore your memory", or the opposite "you can infer this from stuff in the memory - don't re-reason!". There is some work like Dynamic Cheatsheets [1] and Agentic Context Engineering [2] that have studied this aspect, but we are far from a generally reliable solution. And till we have that, I think the system variations for systems trying to solve this problem are going to be (a) LLM-leaning: create unstructured memory files, with human in the loop as a filter to reject inaccurate responses (b) LLM-as-a-layer: what you describe and the article kind of is doing.
[1] https://aclanthology.org/2026.eacl-long.333/ [2] https://openreview.net/pdf?id=eC4ygDs02R
I’ve been thinking about this.. the same idea behind Dynamic cheat sheets paper and the ACE paper. Thanks for those links! Nice to see those ideas academically studied.
My problem is the context of today’s models (that I’ve been using - Claude Opus 5 and GPT-Sol) are a black box to a user like me. I cannot tell what they already hold in their context over the duration of a coding session. I cannot tell influencers a little by using/not using MCP tools, AGENTS.md like MD files, but what they _actually_ use at inference time is still opaque.
For example, if I insist that GPT-Sol use a size bounded dynamic cheat sheet for my project, and even have those reflector&curator agents refine them concurrently, what is to say GPT-Sol doesn’t have an intrinsic cheat sheet like that already, and it is just reading/curating/managing this one in addition and only getting confused needlessly along the way?
Similarly, all those codegraph like tools and semantic search addons.. what is to say the model genuinely uses them instead of relying on its own inference techniques and just “perform” for the user audience as if they are using these fancy tools?
This is not to say we shouldn’t try more ways of managing context and just give up to our AI provider overlords. Just thinking through the logic of what we are working with.
> My problem is the context of today’s models (that I’ve been using - Claude Opus 5 and GPT-Sol) are a black box to a user like me.
Would it even help if it weren’t a blackbox? Maybe to the frontier labs with resources, but even trying to read through the thinking traces of an open model doing a nontrivial task easily takes me an entire day to do correctly (i.e. actually looking at the code it read in) and much more if I’m trying to tune a specific repeatable task.
Applying that to agentic coding seems like a losing proposition. The only way to manage it seems to be more agents and that’s its own rabbit hole of compounding errors (like Claude’s useless comments).