I think this article demonstrates a broader problem with constraint following in LLMs. Agents often satisfy the most obvious and easily verifiable part of a task, but lose track of the constraint that actually determines success.
Testing makes this especially clear: “use fuzzing” turns into generating random bytes, and “use formal verification” into proving a property that isn’t particularly useful. Formally, the technique is applied, but its actual purpose is lost.
This may also explain why different testing techniques produce similar results: the problem may be less about knowing a particular technique and more about the agent’s ability to keep its actual goal as the primary constraint.
the problem may be less about knowing a particular technique and more about the agent’s ability to keep its actual goal as the primary constraint.
I think the limiting factor is often the human's ability to articulate the goal and hold the agent accountable.
The working memory of technical people and the way they communicate often seems to prioritize how they're doing a task and not why they're doing it. So when they talk to you about their problem, they give you the human equivalent of modem noises and stack traces. This is such a common problem on tech support and Q&A forums like StackOverflow that it spawned its own terminology and website: https://xyproblem.info/
The agents don't respond to XY problems any better than a colleague does, and usually much worse.
An effective colleague asks why, and we need a similar relationship between the agent and its human supervisor.
It's well known that CoT needs a narrow problem or several independent narrow problems to perform well. With too broad of a task it often turns into a burden and hurts performance, making it hyperfocus on one thing where a non-reasoning model would just handle everything. I wonder if disabling it would make anything better, but likely not if the model is already trained to do it.
CoT works really well in qwen3.8-flash-next. I think a lot of the problems obversed are literal skill issue: dumping huge system prompts/skills that prime it over some broad scope, when the request is narrow.
The system prompt should be considered a starting point only if you want reusable intelligence and focus. It should not be a grab bag of tools and PR style guides, etc...
Pretty good point - I would actually argue that the imperative nature of prompting takes some of the blame here too. Your initial goal gets "steered" by the agent system and follow-up messages take precedence.
Comments
I think this article demonstrates a broader problem with constraint following in LLMs. Agents often satisfy the most obvious and easily verifiable part of a task, but lose track of the constraint that actually determines success.
Testing makes this especially clear: “use fuzzing” turns into generating random bytes, and “use formal verification” into proving a property that isn’t particularly useful. Formally, the technique is applied, but its actual purpose is lost.
This may also explain why different testing techniques produce similar results: the problem may be less about knowing a particular technique and more about the agent’s ability to keep its actual goal as the primary constraint.
I think the limiting factor is often the human's ability to articulate the goal and hold the agent accountable.
The working memory of technical people and the way they communicate often seems to prioritize how they're doing a task and not why they're doing it. So when they talk to you about their problem, they give you the human equivalent of modem noises and stack traces. This is such a common problem on tech support and Q&A forums like StackOverflow that it spawned its own terminology and website: https://xyproblem.info/
The agents don't respond to XY problems any better than a colleague does, and usually much worse.
An effective colleague asks why, and we need a similar relationship between the agent and its human supervisor.
It's well known that CoT needs a narrow problem or several independent narrow problems to perform well. With too broad of a task it often turns into a burden and hurts performance, making it hyperfocus on one thing where a non-reasoning model would just handle everything. I wonder if disabling it would make anything better, but likely not if the model is already trained to do it.
CoT works really well in qwen3.8-flash-next. I think a lot of the problems obversed are literal skill issue: dumping huge system prompts/skills that prime it over some broad scope, when the request is narrow.
The system prompt should be considered a starting point only if you want reusable intelligence and focus. It should not be a grab bag of tools and PR style guides, etc...
Grab your agent and inspect its prompt.
Pretty good point - I would actually argue that the imperative nature of prompting takes some of the blame here too. Your initial goal gets "steered" by the agent system and follow-up messages take precedence.