Comment on Wikipedia is struggling with voracious AI bot crawlersparentComments−karn971yWhy not just rate limit every user to realistic human rates. You just punish anyone behaving like a bot.−mrweasel1yBecause, as pointed out in another post about the same problem: Many of these scrappers make one or two requests from one IP and then move on.−guerrilla1ySold. Pay by page retrieval rate.
Comments
Why not just rate limit every user to realistic human rates. You just punish anyone behaving like a bot.
Because, as pointed out in another post about the same problem: Many of these scrappers make one or two requests from one IP and then move on.
Sold. Pay by page retrieval rate.