Comment on Robots.txt Disallow: 20 Years of Mistakes To AvoidComments−blueskin_12yThere are enough malicious bots that do follow robots.txt to make it still an important option for most sites.
Comments
There are enough malicious bots that do follow robots.txt to make it still an important option for most sites.