Skip to content

Comment on Ask HN: How are you dealing with scraping hits from EC2 machines?

Comments

If these scrapers abide by robots.txt (you do have a robots.txt, right?) and scrape only whatever you left publicly accessible, then I think you should work on your server performance, because if a scraper causes problems, you'll have much worse problems once whatever you've built becomes popular.

Hunting down bots is a waste of time and effort better spent elsewhere.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.