Single source “massive models” may be more difficult to get out, but Emad said they’re working in licensing a ton of content to train future models. Even then, anyone can train new models now - The output from Dreambooth and Textual Inversion are already impressive, and seem like just the beginning.
Are there known ways the training for these models could be distributed/decomposed? E.g. SETI-style distribution of a homogenous centrally-defined task, or — much more exciting — recombination of several different models / sets of weights? (I'm just throwing words around here without really understanding them.)
I'm imaging a world in which one group of enthusiasts could work together to train a model on all images on Wikipedia, another group could work on training a model that understands hands really well, and then later yet another group could combine the work of the other two without doing all that training from scratch.
The effort and time involved in setting up a distributed community training system would be extremely prone to abuse, errors and uncertainty about the results.
You could get better quality, more quickly by simply running a kickstarter and paying for dedicated gpu time.
not sure how illiterate you are, you're asking good questions, but fwiw if you watch the corridor digital video you should be able to grasp how much transfer learning is possible https://www.youtube.com/watch?v=W4Mcuh38wyM
Just train on the output of existing models minus any photos with watermarks - being twice removed is sure to make it even harder to claim copyright :)
Comments
My take is that the “genie is out of the bottle”
Single source “massive models” may be more difficult to get out, but Emad said they’re working in licensing a ton of content to train future models. Even then, anyone can train new models now - The output from Dreambooth and Textual Inversion are already impressive, and seem like just the beginning.
Going to be an interesting road ahead.
Question from an ML-illiterate:
Are there known ways the training for these models could be distributed/decomposed? E.g. SETI-style distribution of a homogenous centrally-defined task, or — much more exciting — recombination of several different models / sets of weights? (I'm just throwing words around here without really understanding them.)
I'm imaging a world in which one group of enthusiasts could work together to train a model on all images on Wikipedia, another group could work on training a model that understands hands really well, and then later yet another group could combine the work of the other two without doing all that training from scratch.
Is that even remotely plausible?
It’s almost certainly not worth the bother.
The effort and time involved in setting up a distributed community training system would be extremely prone to abuse, errors and uncertainty about the results.
You could get better quality, more quickly by simply running a kickstarter and paying for dedicated gpu time.
not sure how illiterate you are, you're asking good questions, but fwiw if you watch the corridor digital video you should be able to grasp how much transfer learning is possible https://www.youtube.com/watch?v=W4Mcuh38wyM
Yes it is!
Definitely is out of the bottle, especially when training capable cards are getting within reach of regular people.
Sort of hinted at it upthread, but would be interesting if this eventually brings competition to the GPU compute space (AMD, Intel?) .
Just train on the output of existing models minus any photos with watermarks - being twice removed is sure to make it even harder to claim copyright :)