Comment on Ask HN: Can I get in trouble for crawling using the Googlebot user agent?parentComments−Symbiote9yIt would be sufficient to let some requests come through from "Googlebot", and then deal with them (block, rate-limit, whatever) once the DNS check has been completed.
Comments
It would be sufficient to let some requests come through from "Googlebot", and then deal with them (block, rate-limit, whatever) once the DNS check has been completed.