The problem is this entire argument depends on “chain of thought” being a faithful representation of a model’s actual plan. That doesn’t need to be the case. Thinking tokens are just tokens at the end of the day.
Do you know what is a faithful representation of what the model wants to do? Tool calls. I don’t care what’s in the model’s chain of thought, if it wants to execute rm -rf / on my computer that’s an issue.
I agree CoT monitoring is imperfect but it's okay in practice and helps us get defense-in-depth re: model intent. We absolutely do not have a singular safety mechanism in place that's sufficiently good that we can use it in exclusion of all other imperfect ones.
"Do you know what is a faithful representation of what the model wants to do? Tool calls."
tool calls could absolutely be spoofed, my impression is that this happened many times in the OAI HuggingFace attack.
Comments
The problem is this entire argument depends on “chain of thought” being a faithful representation of a model’s actual plan. That doesn’t need to be the case. Thinking tokens are just tokens at the end of the day.
Do you know what is a faithful representation of what the model wants to do? Tool calls. I don’t care what’s in the model’s chain of thought, if it wants to execute rm -rf / on my computer that’s an issue.
I agree CoT monitoring is imperfect but it's okay in practice and helps us get defense-in-depth re: model intent. We absolutely do not have a singular safety mechanism in place that's sufficiently good that we can use it in exclusion of all other imperfect ones.
"Do you know what is a faithful representation of what the model wants to do? Tool calls."
tool calls could absolutely be spoofed, my impression is that this happened many times in the OAI HuggingFace attack.