Ok, after reading through it, I don't think I was speaking inaccurately. Here are some quotes from the blog post (which was very neat, by the way, thanks again)
And it’s part of the lore of neural nets that—in some sense—so long as the setup one has is “roughly right” it’s usually possible to home in on details just by doing sufficient training, without ever really needing to “understand at an engineering level” quite how the neural net has ended up configuring itself.
What determines this structure? Ultimately it’s presumably some “neural net encoding” of features of human language. But as of now, what those features might be is quite unknown. In effect, we’re “opening up the brain of ChatGPT” (or at least GPT-2) and discovering, yes, it’s complicated in there, and we don’t understand it—even though in the end it’s producing recognizable human language.
This is the kind of thing I'm referring to. Even though we can look at pictures of the neuron activations, we don't really know what it's doing, any more than you can look at a picture of a brain scan of a person speaking and know why they decided to say those particular words. We know at a low level how the network works, of course, because we coded it. It's the emergent behavior that we don't understand, and that's the bit that's scary because that's where the AI risk lives. It's like how we understand particle physics pretty well, and chemistry to some degree, but biology is a massively complicated jungle that we've barely scratched the surface of.
Maybe there's something we could develop analogous to using an MRI as a lie detector, for these networks. But as far as I know, this is still an unsolved problem, and apparently really hard for networks that are smarter than you are: https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H...
Comments
Ok, after reading through it, I don't think I was speaking inaccurately. Here are some quotes from the blog post (which was very neat, by the way, thanks again)
This is the kind of thing I'm referring to. Even though we can look at pictures of the neuron activations, we don't really know what it's doing, any more than you can look at a picture of a brain scan of a person speaking and know why they decided to say those particular words. We know at a low level how the network works, of course, because we coded it. It's the emergent behavior that we don't understand, and that's the bit that's scary because that's where the AI risk lives. It's like how we understand particle physics pretty well, and chemistry to some degree, but biology is a massively complicated jungle that we've barely scratched the surface of.
Maybe there's something we could develop analogous to using an MRI as a lie detector, for these networks. But as far as I know, this is still an unsolved problem, and apparently really hard for networks that are smarter than you are: https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H...