Comment on Amazon has a way to scrape GitHub and feed its AI modelparentComments−graemep2yI think that is very different from cloning repos from Github in accordance with the licences.I agree crawling sites to the extent it causes problems is a problem.Googlebot and Bingbot do follow robots.txt and respect HTTP 429 responses and usually have reasonable default crawl rates.Is it possible that these are scrapers using fake UA strings?−73737373732yIs it possible that these are scrapers using fake UA strings?I now believe this is the case−nextaccountic2yYou need to check if the requests come from Googlebot's IP rangeshttps://developers.google.com/search/docs/crawling-indexing/...https://searchengineland.com/google-publishes-the-list-of-go...https://developers.google.com/search/apis/ipranges/googlebot...
Comments
I think that is very different from cloning repos from Github in accordance with the licences.
I agree crawling sites to the extent it causes problems is a problem.
Googlebot and Bingbot do follow robots.txt and respect HTTP 429 responses and usually have reasonable default crawl rates.
Is it possible that these are scrapers using fake UA strings?
I now believe this is the case
You need to check if the requests come from Googlebot's IP ranges
https://developers.google.com/search/docs/crawling-indexing/...
https://searchengineland.com/google-publishes-the-list-of-go...
https://developers.google.com/search/apis/ipranges/googlebot...