Well put. The NLP community seems to be slowly standardizing on language models trained on large unlabeled corpora as an analogue to pre-trained ImageNet models (ELMo, ULMFit, "Improving Language Understanding by Generative Pre-training"). The gradient publication does a good job of detailing this: https://thegradient.pub/nlp-imagenet/.
Recent research is finally checking off a few important boxes that are required for widespread applicability:
- Minimal configuration required
Aside from tweaking the language modeling loss coefficient language model finetuning seems to "just work". ULMFiT's approach also requires minimal configuration.
- Reasonable training times
You can finetune these transformer models on a few hundred examples in 10 minutes on a single GPU.
- Beneficial with very small amounts of labeled training data
This approach consistently beats out the use of pretrained word/document embeddings at ~200 training examples. Will be posting some benchmarks on two dozen classification tasks in the near future.
There are a few remaining conditions that I think need to be met before this kind of approach sees widespread use:
- Reasonable inference times
Inference is still rather slow because of model complexity.
- Reasonable memory consumption
Transfer learning is typically well suited to personalization tasks because of limited training data requirements, but large memory footprints mean that it's hard to swap out models for different users on the fly.
Comments
Well put. The NLP community seems to be slowly standardizing on language models trained on large unlabeled corpora as an analogue to pre-trained ImageNet models (ELMo, ULMFit, "Improving Language Understanding by Generative Pre-training"). The gradient publication does a good job of detailing this: https://thegradient.pub/nlp-imagenet/.
Recent research is finally checking off a few important boxes that are required for widespread applicability:
- Minimal configuration required
Aside from tweaking the language modeling loss coefficient language model finetuning seems to "just work". ULMFiT's approach also requires minimal configuration.
- Reasonable training times
You can finetune these transformer models on a few hundred examples in 10 minutes on a single GPU.
- Beneficial with very small amounts of labeled training data
This approach consistently beats out the use of pretrained word/document embeddings at ~200 training examples. Will be posting some benchmarks on two dozen classification tasks in the near future.
There are a few remaining conditions that I think need to be met before this kind of approach sees widespread use:
- Reasonable inference times
Inference is still rather slow because of model complexity.
- Reasonable memory consumption
Transfer learning is typically well suited to personalization tasks because of limited training data requirements, but large memory footprints mean that it's hard to swap out models for different users on the fly.