You do not need the original dataset to train the model on an additional dataset
Maybe I misunderstood your original question. To be clear, the process of modifying a trained model does not require the presence of the original data. You said “deleted” which perhaps I misinterpreted. You’re not “instantiating a new model from scratch” when you modify it. You’re continuing to train it where it left off.
What if you want to start with a subset of the original data? Like you've trained a model, and then later said "You know, this new data we're adding is great, but maybe pulling all those comments from 4chan earlier was a mistake," wouldn't that require starting fresh with access to the actual data?
Comments
You do not need the original dataset to train the model on an additional dataset
Maybe I misunderstood your original question. To be clear, the process of modifying a trained model does not require the presence of the original data. You said “deleted” which perhaps I misinterpreted. You’re not “instantiating a new model from scratch” when you modify it. You’re continuing to train it where it left off.
What if you want to start with a subset of the original data? Like you've trained a model, and then later said "You know, this new data we're adding is great, but maybe pulling all those comments from 4chan earlier was a mistake," wouldn't that require starting fresh with access to the actual data?
Technically correct but not a very realistic request / approach.
The general idea is to get as good of a mastery of language as possible, generally, and then fine tune to specialize on tasks