Because LLMs degrade with context length. And even if they didn't, having to constantly ingest the same stuff is wasteful for execution and cost. Why stuff the context when you don't have to?
Conservatively speaking, LLMs degrade past 40% of a 2M context window, 4k tokens is 0.2%, so no degradation there. Thats also current generation conservative estimate.
I think wasteful is an irrelevant metric. Claude ingests those tokens in a quarter of a second, if it causes it to catch any bug ever it saves far more time than it ever uses.
Any individual default that causes unexpected behavior causes more problems, takes more time and costs more than the small cost of being explicit.
Comments
Because LLMs degrade with context length. And even if they didn't, having to constantly ingest the same stuff is wasteful for execution and cost. Why stuff the context when you don't have to?
Conservatively speaking, LLMs degrade past 40% of a 2M context window, 4k tokens is 0.2%, so no degradation there. Thats also current generation conservative estimate.
I think wasteful is an irrelevant metric. Claude ingests those tokens in a quarter of a second, if it causes it to catch any bug ever it saves far more time than it ever uses.
Any individual default that causes unexpected behavior causes more problems, takes more time and costs more than the small cost of being explicit.
Boiling frog problem
the Electron problem