Comment on Amazon has a way to scrape GitHub and feed its AI modelparentComments−73737373732yIs it possible that these are scrapers using fake UA strings?I now believe this is the case−nextaccountic2yYou need to check if the requests come from Googlebot's IP rangeshttps://developers.google.com/search/docs/crawling-indexing/...https://searchengineland.com/google-publishes-the-list-of-go...https://developers.google.com/search/apis/ipranges/googlebot...
Comments
I now believe this is the case
You need to check if the requests come from Googlebot's IP ranges
https://developers.google.com/search/docs/crawling-indexing/...
https://searchengineland.com/google-publishes-the-list-of-go...
https://developers.google.com/search/apis/ipranges/googlebot...