Being one of the most downloaded datasets on Huggingface, I was a little bit surprised by how dirty this dataset was. Plus it had very limited information and some incorrect classifications as well.
For an internal experiment on building a "Truthful Evaluator", we picked up this dataset and tried fine-tuning a model on these 8000 odd examples.
Realised that it needed:
1. Cleaning up
2. Some reclassification
But, most importantly - it lacked context data. It only had a link pointing to the source which was also absent for a few rows.
We scraped the internet for the link in the dataset, matched it to the question and narrowed down on a small context to be added to the main dataset.
Releasing it publicly so that someone else may avoid the 2-3 days of pain of wrangling with this data.
Comments
Being one of the most downloaded datasets on Huggingface, I was a little bit surprised by how dirty this dataset was. Plus it had very limited information and some incorrect classifications as well.
For an internal experiment on building a "Truthful Evaluator", we picked up this dataset and tried fine-tuning a model on these 8000 odd examples.
Realised that it needed: 1. Cleaning up 2. Some reclassification
But, most importantly - it lacked context data. It only had a link pointing to the source which was also absent for a few rows.
We scraped the internet for the link in the dataset, matched it to the question and narrowed down on a small context to be added to the main dataset.
Releasing it publicly so that someone else may avoid the 2-3 days of pain of wrangling with this data.