1. CoT is a pretty mediocre debugging proxy due to multiple phenomena, even if you train it for readability. You can have some direct intuition about how misleading it is by trying to hijack it to follow a fixed plan. Having a trace is useful in many cases but it's very far from reading model's intent.
2. Nothing prevents you from probing and interpreting the state directly if you want. Passing everything through the token transport is not really required for that.
Comments
This reads like typical LW FUD.
1. CoT is a pretty mediocre debugging proxy due to multiple phenomena, even if you train it for readability. You can have some direct intuition about how misleading it is by trying to hijack it to follow a fixed plan. Having a trace is useful in many cases but it's very far from reading model's intent.
2. Nothing prevents you from probing and interpreting the state directly if you want. Passing everything through the token transport is not really required for that.