A lot of the corpus feeding GPT-3 was web scraping and now that source is tainted for future GPT-x models
It might be possible to filter out GPT-3 generated text from future training data. Simply feed part of the text to GPT-3, and if it is way too good at predicting what follows you throw it out. The same trick could be used to detect students writing essays with it.
This trick will stop working though as more variants of good text prediction algorithms appear, unless we can do the same test against each one.
If you wanted to generate a longer essay by GPT3 but make it really hard to tell, you could just get it started with a 1-2 sentences. Then let it generate N-1 tokens (where N is the window size, I guess 2048), and write a few tokens yourself (and repeat). Then I don't think there's any way to reverse engineer the internal state of the model at any point in the text. It would be like trying to find a string that hashes to a given value.
There is one fringe ironic and probably unrealistic failure case with that approach - what if GPT3 matches expectations too well with actual people and results in good examples being tossed in preference for posts which don't fit with most like say blatant spambots?
Comments
It might be possible to filter out GPT-3 generated text from future training data. Simply feed part of the text to GPT-3, and if it is way too good at predicting what follows you throw it out. The same trick could be used to detect students writing essays with it.
This trick will stop working though as more variants of good text prediction algorithms appear, unless we can do the same test against each one.
If you wanted to generate a longer essay by GPT3 but make it really hard to tell, you could just get it started with a 1-2 sentences. Then let it generate N-1 tokens (where N is the window size, I guess 2048), and write a few tokens yourself (and repeat). Then I don't think there's any way to reverse engineer the internal state of the model at any point in the text. It would be like trying to find a string that hashes to a given value.
Maybe like hashing though there can be weaknesses in the algorithm still.
There is one fringe ironic and probably unrealistic failure case with that approach - what if GPT3 matches expectations too well with actual people and results in good examples being tossed in preference for posts which don't fit with most like say blatant spambots?