As we describe in our paper, we can use these same techniques to change the strength of safety-related features—like those related to dangerous computer code, criminal activity, or deception. With further research, we believe this work could help make AI models safer.
Does this mean the corollary is also true? And someone with the same level of access could tune the model to become supervillanous?
Absolutely. "Refusals" all seem to follow a similar direction in the LLM residual stream, so preventing the model from representing this direction prevents it from refusing.
Hopefully this will make it easy to customize models for all of the different personal value systems on earth rather than these companies trying to project their single value system on all of us. I see this as a type of localization: at some point you have to acknowledge that the software you make is being used by people who are different than you and have different expectations.
Even the topic of “criminal activity” will not be the same from jurisdiction to jurisdiction so the model will need to have some contextual awareness and ability to tailor its responses appropriately.
It must. IIRC Anthropic has a 'red team' of sorts. I wonder what they can do with this technique? What are the limits of "evil" of these current models?
It's good if you really don't want your LLM to mention specific things, which I can see some groups wanting. Having it mention some things even when they're not related could be good for integrated ads in a chatbot, which sounds evil in that it would be really annoying. Your friend's account gets hacked, a chatbot LLM is finetuned on their message history, it's able to carry on a conversation while slipping in a mention of Joe's Hot Dogs every now and then.
It probably also could help with consistency when trying to do LangChain-type stuff.
Comments
Does this mean the corollary is also true? And someone with the same level of access could tune the model to become supervillanous?
Absolutely. "Refusals" all seem to follow a similar direction in the LLM residual stream, so preventing the model from representing this direction prevents it from refusing.
https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in...
Hopefully this will make it easy to customize models for all of the different personal value systems on earth rather than these companies trying to project their single value system on all of us. I see this as a type of localization: at some point you have to acknowledge that the software you make is being used by people who are different than you and have different expectations.
Even the topic of “criminal activity” will not be the same from jurisdiction to jurisdiction so the model will need to have some contextual awareness and ability to tailor its responses appropriately.
It must. IIRC Anthropic has a 'red team' of sorts. I wonder what they can do with this technique? What are the limits of "evil" of these current models?
It's good if you really don't want your LLM to mention specific things, which I can see some groups wanting. Having it mention some things even when they're not related could be good for integrated ads in a chatbot, which sounds evil in that it would be really annoying. Your friend's account gets hacked, a chatbot LLM is finetuned on their message history, it's able to carry on a conversation while slipping in a mention of Joe's Hot Dogs every now and then.
It probably also could help with consistency when trying to do LangChain-type stuff.