Comment on Robots.txt Disallow: 20 Years of Mistakes To AvoidComments−sp33212yWhy does Google ignore the crawl delay?−sbierwagen12yGoogle has millions of spiders, in datacenters all over the world. Maybe respecting crawl delay added more shared-state overhead than they wanted.−jhwhite12yI haven't read the entire article but we were discussing this at work a few weeks ago. You can set the crawl delay in Google Web Master Tools but they only adhere to that setting for 90 days then they go back to their default.
Comments
Why does Google ignore the crawl delay?
Google has millions of spiders, in datacenters all over the world. Maybe respecting crawl delay added more shared-state overhead than they wanted.
I haven't read the entire article but we were discussing this at work a few weeks ago. You can set the crawl delay in Google Web Master Tools but they only adhere to that setting for 90 days then they go back to their default.