Skip to content

Comment on Matt Cutts is looking for scraper sitesparent

Comments

You can tell google not to download your site with your robots.txt file.

Also google can impose whatever rules they fancy, because it's their own site. Do you have a website? If I find some that you govern it with some rule that I don't like, should I rage on forums about it? Should you care if I rage about it?

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.