I don't think this is very far from innovating. The foundation is building up an accurate mental model of the world, especially in the area you want to innovate in, and then running thought experiments, following through them step by step, and testing in the real world to make sure you're not straying too far. I think being able to write a new, coherent essay involves a lot of the same skills.
But the one building the coherent essay is the author of the article, not GPT-3. The author of the article is taking GPT-3 output that sounds good to them, and stitching it together into an essay.
Also note, the one giving the words meaning in the first place is you. GPT-3 is simply repeating patterns that it discovered, but it doesn't have any notion of meaning, any model of the world outside "this is what text I expect to see following this text".
I am not merely being philosophical here. Two GPT-3 instances couldn't "teach" each other even the slightest bit of new information, or prime each other to get some specific kind of response, perhaps trying a few different things to see which prompts are more likely to produce the desired outcome in the other instance - because there is no "desired outcome" - it's all just trying (and usually succeeding exceedingly well!) to produce text that sounds like text it has seen before, while matching the prompt.
You may be right, I honestly don't know how this particular algorithm works. We may be closer to true AI intelligence than I think, but I am not sure anyone should wish for that. I am certain that we are still decades away from true human intelligence (Einstein level), and maybe a few years to a decade away from fully replacing the cognitive abilities of an average human. But it will come. And I am not sure the human race will be able to survive it.
I have no more evidence than you do, so take it with a grain of salt; but I think you're mistaken on the intelligence axis. As in, I think the distance from GPT-3 right now to average human, is way larger than from avg human to Einstein. And I agree we're not that far from replacing the work of an average human.
I think I agree with those points. However, the most interesting thing about GPT-3 in my mind is how it was largely a naive scaling up of GPT-2.
I would be very surprised to not see a GPT-4-like system built in the next several years by simply scaling up by 100x again (perhaps with some algorithmic improvements thrown in as well). I don’t know how it will perform but I expect it to be significantly more impressive than GPT-3. Which would be a very interesting thing... the world would look very different. Potentially even scary, especially with immediate follow-up reapplications of the techniques in other areas.
Comments
I don't think this is very far from innovating. The foundation is building up an accurate mental model of the world, especially in the area you want to innovate in, and then running thought experiments, following through them step by step, and testing in the real world to make sure you're not straying too far. I think being able to write a new, coherent essay involves a lot of the same skills.
But the one building the coherent essay is the author of the article, not GPT-3. The author of the article is taking GPT-3 output that sounds good to them, and stitching it together into an essay.
Also note, the one giving the words meaning in the first place is you. GPT-3 is simply repeating patterns that it discovered, but it doesn't have any notion of meaning, any model of the world outside "this is what text I expect to see following this text".
I am not merely being philosophical here. Two GPT-3 instances couldn't "teach" each other even the slightest bit of new information, or prime each other to get some specific kind of response, perhaps trying a few different things to see which prompts are more likely to produce the desired outcome in the other instance - because there is no "desired outcome" - it's all just trying (and usually succeeding exceedingly well!) to produce text that sounds like text it has seen before, while matching the prompt.
You may be right, I honestly don't know how this particular algorithm works. We may be closer to true AI intelligence than I think, but I am not sure anyone should wish for that. I am certain that we are still decades away from true human intelligence (Einstein level), and maybe a few years to a decade away from fully replacing the cognitive abilities of an average human. But it will come. And I am not sure the human race will be able to survive it.
I have no more evidence than you do, so take it with a grain of salt; but I think you're mistaken on the intelligence axis. As in, I think the distance from GPT-3 right now to average human, is way larger than from avg human to Einstein. And I agree we're not that far from replacing the work of an average human.
I think I agree with those points. However, the most interesting thing about GPT-3 in my mind is how it was largely a naive scaling up of GPT-2.
I would be very surprised to not see a GPT-4-like system built in the next several years by simply scaling up by 100x again (perhaps with some algorithmic improvements thrown in as well). I don’t know how it will perform but I expect it to be significantly more impressive than GPT-3. Which would be a very interesting thing... the world would look very different. Potentially even scary, especially with immediate follow-up reapplications of the techniques in other areas.