Comment on Robots.txt Disallow: 20 Years of Mistakes To AvoidComments−dredge12yThe article contains some good observations, but I'm struggling to understand this one:"Some sites try to communicate with Google through comments in robots.txt"In the examples given, none appear to be trying to "communicate with Google through comments" - how is including... # What's all this then? # \ # # ----- # | . . | # ----- # \--|-|--/ # | | # |-------| ...a "mistake" to avoid? There's no harm in it at all.−Istof12y"Some sites try to communicate with Google through comments in robots.txt"I thought that was the whole point of robots.txt−lmm12yNo, the point is to communicate with Google through non-comments in robots.txt.−SoftwareMaven12yI don't think those examples were of people trying to communicate with a crawler. I think they were examples of comments that the owners knew would be thrown away by crawlers.
Comments
The article contains some good observations, but I'm struggling to understand this one:
"Some sites try to communicate with Google through comments in robots.txt"
In the examples given, none appear to be trying to "communicate with Google through comments" - how is including...
...a "mistake" to avoid? There's no harm in it at all."Some sites try to communicate with Google through comments in robots.txt"
I thought that was the whole point of robots.txt
No, the point is to communicate with Google through non-comments in robots.txt.
I don't think those examples were of people trying to communicate with a crawler. I think they were examples of comments that the owners knew would be thrown away by crawlers.