Skip to content

Comment on Ask HN: A good list of bots/crawlers ip addresses?

Comments

Lists like this aren't generally shared, because then the nefarious bots would know they had been caught.

Well behaved bots tend to use useragents that make themselves fairly obvious.

The best bet is to watch your logs for an IP or agent that seems to hit more URLs than anyone else, and then investigate by hand.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.