Comment on Ask HN: Can I get in trouble for crawling using the Googlebot user agent?parentComments−lend0009yUnless you do it from a Google Cloud instance, that is.−mootothemax9yUnless you do it from a Google Cloud instance, that is.What's the reverse DNS for Google Cloud IPs? Google says to check that Googlebot's IP resolves to either a .google.com or .googlebot.com domain.https://support.google.com/webmasters/answer/80553?hl=en−milankragujevic9yCouldn't you use GWT Mobilizer to scrape a site then index that?Like this: http://i.imgur.com/ocR54Yq.jpg−lend0009yGood point -- although it makes sense why it isn't frequently implemented. DNS lookups aren't cheap for this kind of thing.−Symbiote9yIt would be sufficient to let some requests come through from "Googlebot", and then deal with them (block, rate-limit, whatever) once the DNS check has been completed.−hrrsn9y.googleusercontent.com
Comments
Unless you do it from a Google Cloud instance, that is.
What's the reverse DNS for Google Cloud IPs? Google says to check that Googlebot's IP resolves to either a .google.com or .googlebot.com domain.
https://support.google.com/webmasters/answer/80553?hl=en
Couldn't you use GWT Mobilizer to scrape a site then index that?
Like this: http://i.imgur.com/ocR54Yq.jpg
Good point -- although it makes sense why it isn't frequently implemented. DNS lookups aren't cheap for this kind of thing.
It would be sufficient to let some requests come through from "Googlebot", and then deal with them (block, rate-limit, whatever) once the DNS check has been completed.
.googleusercontent.com