Actually that point isn't necessarily obvious to me. Maybe we can create an AI which is 10% smarter than ourselves, but maybe a 10% smarter AI still isn't smart enough to create an AI 10% smarter than itself. Maybe building a 121% AI that smart is such a hard task that even our 110% AI can't manage it -- maybe the best it can do is a 115% AI.
Analogy: Out of Lego, I could build a robot arm that's capable of building a slightly larger robot arm out of Lego. But I can't just keep on doing this 'til I have a robot arm the size of the Burj Dubai, because there are limits to how big a Lego-based robot arm can be before it collapses under its own weight.
I'm not saying that AIs are like this, just that they might be.
Your analogy is particularly apt if you consider that two of the means of achieving AI are emergent and thus not engineering and not at all friendly to reverse engineering. If we can simulate the brain's processes in bulk, we still don't necessarily understand how it does what it does. (Like making a photocopy of ciphertext.) Likewise, if we create some sort of architecture that lets us evolve AIs, there is no guarantee that we will be able to understand how the results work. There's no guarantee that the resulting AIs will be able to either.
Comments
Actually that point isn't necessarily obvious to me. Maybe we can create an AI which is 10% smarter than ourselves, but maybe a 10% smarter AI still isn't smart enough to create an AI 10% smarter than itself. Maybe building a 121% AI that smart is such a hard task that even our 110% AI can't manage it -- maybe the best it can do is a 115% AI.
Analogy: Out of Lego, I could build a robot arm that's capable of building a slightly larger robot arm out of Lego. But I can't just keep on doing this 'til I have a robot arm the size of the Burj Dubai, because there are limits to how big a Lego-based robot arm can be before it collapses under its own weight.
I'm not saying that AIs are like this, just that they might be.
Your analogy is particularly apt if you consider that two of the means of achieving AI are emergent and thus not engineering and not at all friendly to reverse engineering. If we can simulate the brain's processes in bulk, we still don't necessarily understand how it does what it does. (Like making a photocopy of ciphertext.) Likewise, if we create some sort of architecture that lets us evolve AIs, there is no guarantee that we will be able to understand how the results work. There's no guarantee that the resulting AIs will be able to either.
I think Godel's Incompleteness Theorm implies the machines can't grok the code they are made from. I'm not sure though.