Comment on Ask HN: A good list of bots/crawlers ip addresses?Comments−jedberg15yLists like this aren't generally shared, because then the nefarious bots would know they had been caught.Well behaved bots tend to use useragents that make themselves fairly obvious.The best bet is to watch your logs for an IP or agent that seems to hit more URLs than anyone else, and then investigate by hand.
Comments
Lists like this aren't generally shared, because then the nefarious bots would know they had been caught.
Well behaved bots tend to use useragents that make themselves fairly obvious.
The best bet is to watch your logs for an IP or agent that seems to hit more URLs than anyone else, and then investigate by hand.