Skip to content

Comment on Ask HN: Can I get in trouble for crawling using the Googlebot user agent?

Comments

One thing they could do is tell you to stop. If they have told you to stop, and taken measures to block you out (blocking crawlers besides google) then persisting is illegal. I believe somebody got succesfully sued by facebook for continuing to scrape after facebook told them to stop. I'm not sure about the legality if you haven't been explicitly asked to stop, but as long as you are never blatant enough for them to notice, you shouldn't have any trouble.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.