Comment on Ask HN: Can I get in trouble for crawling using the Googlebot user agent?parentComments−mathrawka9yNot to detract from your point, but blindly blocking a User Agent because of a bad actor and losing money is not a good solution.A more robust solution can be coded using information from Google: https://support.google.com/webmasters/answer/80553?hl=en
Comments
Not to detract from your point, but blindly blocking a User Agent because of a bad actor and losing money is not a good solution.
A more robust solution can be coded using information from Google: https://support.google.com/webmasters/answer/80553?hl=en