Skip to content

Comment on Messing with scraper botsparent

Comments

It's different. I'm fine with someone scraping my website as a good citizen, by identifying themselves in their user-agent string and preferably respecting robots.txt. I'm not, however, fine with tens of requests per second to every possible URL from random IPs I'm receiving right now, all pretending to be different old versions of Chrome.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.