Someone pointed this out on Twitter - it looks like StackOverflow recently updated their robots.txt file to explicitly disallow all crawlers. Obviously, this won't stop those that don't respect robots.txt, but I found this decision strange. Not even Google or Bing's crawlers (which respect robots.txt) will be able to crawl StackOverflow, which could be the final nail in the coffin for SO, since (presumably) most StackOverflow traffic comes from search engines.
Comments
Someone pointed this out on Twitter - it looks like StackOverflow recently updated their robots.txt file to explicitly disallow all crawlers. Obviously, this won't stop those that don't respect robots.txt, but I found this decision strange. Not even Google or Bing's crawlers (which respect robots.txt) will be able to crawl StackOverflow, which could be the final nail in the coffin for SO, since (presumably) most StackOverflow traffic comes from search engines.
https://meta.stackexchange.com/questions/404687/our-robots-t...
I think this still works for crawlers like Google
I just tested it with ChatGPT agent mode and it returned a 502 bad gateway, so even agents are not going to be a source of traffic.
They must really want only organic traffic for some reason.