Google has millions of spiders, in datacenters all over the world. Maybe respecting crawl delay added more shared-state overhead than they wanted.