Is there any chance we might see an updated index for the Common Crawl? I've tried using the dataset before, but I found it difficult given that you have to process the entire thing in order to find the particular pages you are looking for.
As an example, I was trying a project to look at the top news sites, like BBC, CNN, Al Jazeera, etc. Then matching articles about the same news topic on each site, before finally fact checking for differences between the stories (ie. 20,000 homes were without power vs. 50,000)
That kind of project requires a load of crawling, but I can't look for specific pages without processing the entire CC set first.
I love the project though, so thank you for doing it!
Comments
Is there any chance we might see an updated index for the Common Crawl? I've tried using the dataset before, but I found it difficult given that you have to process the entire thing in order to find the particular pages you are looking for.
As an example, I was trying a project to look at the top news sites, like BBC, CNN, Al Jazeera, etc. Then matching articles about the same news topic on each site, before finally fact checking for differences between the stories (ie. 20,000 homes were without power vs. 50,000)
That kind of project requires a load of crawling, but I can't look for specific pages without processing the entire CC set first.
I love the project though, so thank you for doing it!