Comment on StackOverflow disallows all crawlers in robots.txt fileparentComments−integralid1yhttps://meta.stackexchange.com/questions/404687/our-robots-t...After this change, any trusted crawler will continue to see the old copy of robots.txt.I think this still works for crawlers like Google
Comments
https://meta.stackexchange.com/questions/404687/our-robots-t...
I think this still works for crawlers like Google