Skip to content

Comment on Ask HN: Kind of data needed to build a language model?

Comments

See here for some machine translation training data sets

https://metatext.io/datasets-list/translation-task

they might not have what you're looking for but you'll probably need something similar to one of the data sets they have.

Most systems now are trained on parallel corpuses, for instance there is a collection of 30,000 sentences in English and Japanese listed on that site. If you've got enough training examples you don't need a dictionary, a specification of the grammar or anything else. You need a lot of text though.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.