Comment on 3taps says CL blocking "all general search engines"parentComments−mooism214yGoogle, and I presume other big search engines too, cache robots.txt for a week by default. We're well within the window for them to still be indexing CL.−sigmadeltaOP14yIf that's true, how does 3taps know that CL is blocking search engines? Also:$ wget -q -O- --save-headers http://www.craigslist.org/robots.txt | fgrep Last-ModifiedLast-Modified: Fri, 04 Nov 2011 18:13:24 GMT
Comments
Google, and I presume other big search engines too, cache robots.txt for a week by default. We're well within the window for them to still be indexing CL.
If that's true, how does 3taps know that CL is blocking search engines? Also:
$ wget -q -O- --save-headers http://www.craigslist.org/robots.txt | fgrep Last-Modified
Last-Modified: Fri, 04 Nov 2011 18:13:24 GMT