Comment on Dawn of a new era in Search: Balancing innovation, competition, and public goodparentComments−1vuio0pswjnm72yDoes CC publish the methodology for how they determine what to crawl. More particularly, how do they determine what not to crawl.−arnaudsm2yYes a few big sites are missing, notably reddit. Most of CC is spam though, the real useful content is really small.I'm experimenting my own search engine at the moment, and am considering to make it public at some point. It's not that impossible of a task!
Comments
Does CC publish the methodology for how they determine what to crawl. More particularly, how do they determine what not to crawl.
Yes a few big sites are missing, notably reddit. Most of CC is spam though, the real useful content is really small.
I'm experimenting my own search engine at the moment, and am considering to make it public at some point. It's not that impossible of a task!