I disagree with this. Decoders were absolutely dominant in 2020 for chat. GPT2 was considered too dangerous to release, and I remember scrambling to get on the GPT3 waitlist. It worked.
He's not talking about decoders, he's talking about auto-regression. Before ChatGPT, the dominant paradigm was fine-tuning BERT-like models.
Before ChatGPT there really wasn’t much of a concept of pre-training and post-training.
Again, people spend years just post-training BERTs in various ways.
It’s both. Fine-tuning a BERT model changes its weights, which causes the embeddings to change.
For example you might have one model which embeds a text query and another model which embeds an image. You also have a dataset of image + text captions. Training means updating the weights of those models so that the embedding of the image is close to the embedding of its corresponding caption.
Comments
He's not talking about decoders, he's talking about auto-regression. Before ChatGPT, the dominant paradigm was fine-tuning BERT-like models.
Again, people spend years just post-training BERTs in various ways.
GPT2 and 3 work via autoregression. In chat bots, decoders work via autoregression.
Yes, I was one of them. That’s not called “post-training” it’s called fine-tuning.
Asking out of curiosity, because I have limited experience in that domain.
I thought that fine-tuning was changing the weights in the model, not the embedding? Or did I misunderstand?
It’s both. Fine-tuning a BERT model changes its weights, which causes the embeddings to change.
For example you might have one model which embeds a text query and another model which embeds an image. You also have a dataset of image + text captions. Training means updating the weights of those models so that the embedding of the image is close to the embedding of its corresponding caption.