Comment on Ask HN: 16 yo Nephew, in E. Africa, wants to train an LLM with on disk WikipediaComments−FrenchDevRemote2yNot an expert, but maybe using RAG/embeddings on the on-disk wikipedia would be better than finetuning on wikipedia?Most decent LLMs probably were already trained on wikipedia, that doesn't stop them from hallucinating when asked questions about it.−a_wOP2yThanks for the suggestion! I will look into this.−more_corn2y^ This is the way
Comments
Not an expert, but maybe using RAG/embeddings on the on-disk wikipedia would be better than finetuning on wikipedia?
Most decent LLMs probably were already trained on wikipedia, that doesn't stop them from hallucinating when asked questions about it.
Thanks for the suggestion! I will look into this.
^ This is the way