One of my sites has been spammed by scrapers (Bytedance's Bytespider, Googlebot, Bingbot) several thousand times within just an hour, to the point of making it break. They do this without notification or asking for consent of the users creating the content they ingest and possibly use to train AI models with, and also without credit or compensation. I think the world needs strict regulation against this kind of parasitic, likely illegal behavior.
1994 called and wants your opinion on (hot)linking.
If you email them, they'll usually respond quickly and stop. Otherwise, what's stopping you from blocking / rate limiting their ranges?
I set them up over a decade ago, but I still have some honeypot crawler traps just to keep their cores busy ingesting junk, with autoupdating rules for some of my domains. If they're not obligated to ask, I'm not obligated to give them anything useful.
19010 requests from user agents "Googlebot" and "bingbot" combined within the last 24h. Bytespider has died down to 74. 17 from Claudebot. But I've heard from a HN user that Bytedance just change their user agent when they are blocked. I'm blocking all of them with Cloudflare.
Comments
One of my sites has been spammed by scrapers (Bytedance's Bytespider, Googlebot, Bingbot) several thousand times within just an hour, to the point of making it break. They do this without notification or asking for consent of the users creating the content they ingest and possibly use to train AI models with, and also without credit or compensation. I think the world needs strict regulation against this kind of parasitic, likely illegal behavior.
1994 called and wants your opinion on (hot)linking.
If you email them, they'll usually respond quickly and stop. Otherwise, what's stopping you from blocking / rate limiting their ranges?
I set them up over a decade ago, but I still have some honeypot crawler traps just to keep their cores busy ingesting junk, with autoupdating rules for some of my domains. If they're not obligated to ask, I'm not obligated to give them anything useful.
I think that is very different from cloning repos from Github in accordance with the licences.
I agree crawling sites to the extent it causes problems is a problem.
Googlebot and Bingbot do follow robots.txt and respect HTTP 429 responses and usually have reasonable default crawl rates.
Is it possible that these are scrapers using fake UA strings?
I now believe this is the case
You need to check if the requests come from Googlebot's IP ranges
https://developers.google.com/search/docs/crawling-indexing/...
https://searchengineland.com/google-publishes-the-list-of-go...
https://developers.google.com/search/apis/ipranges/googlebot...
Did they ignore your robots.txt file?
I changed it a few days ago, still getting hit by over 14800 requests within 24 hours
From google etc? Sounds like an abuse case
19010 requests from user agents "Googlebot" and "bingbot" combined within the last 24h. Bytespider has died down to 74. 17 from Claudebot. But I've heard from a HN user that Bytedance just change their user agent when they are blocked. I'm blocking all of them with Cloudflare.