The biggest difficulty people have IMO when trying to train language models in non-English languages is that there is not enough text written in these other languages to select a big good quality dataset.
Also, there are lots of (poorly) machine translated websites in Polish... So any dataset that contains web crawl will have precisely what I'd prefer not to have.
Ideally I'd see either the national gov, or the EU to invest money into creation of more high quality datasets in all EU languages.
So when I point out the failures of for example chatgpt in my language I do so while being amazed it can generate and understand Polish at all.
Also regarding multilingual model size being larger. I never heard this before, but it seems logical. I have heard models gain extra performance on English tasks when they are trained on other languages too so there is a benefit to adding multilingual datasets.
Comments
The biggest difficulty people have IMO when trying to train language models in non-English languages is that there is not enough text written in these other languages to select a big good quality dataset.
Also, there are lots of (poorly) machine translated websites in Polish... So any dataset that contains web crawl will have precisely what I'd prefer not to have.
Ideally I'd see either the national gov, or the EU to invest money into creation of more high quality datasets in all EU languages.
So when I point out the failures of for example chatgpt in my language I do so while being amazed it can generate and understand Polish at all.
Also regarding multilingual model size being larger. I never heard this before, but it seems logical. I have heard models gain extra performance on English tasks when they are trained on other languages too so there is a benefit to adding multilingual datasets.