Comment on HuggingFace: Security.txtparentComments−cgannett1dRobots.txt became 0% effective eventually. But this? With the way LLMs work? You never know.−dguest1dClaude seems to follow robots.txt by default. Actually at my organization our theory is that this is why no one is finding our public results any more.−Den_VR1dPeople still rail against Robots.txt crimes, to the point of self destructing all their own content.−warkdarrior1dI am adding it to my instruction-following training data, as a negative sample.
Comments
Robots.txt became 0% effective eventually. But this? With the way LLMs work? You never know.
Claude seems to follow robots.txt by default. Actually at my organization our theory is that this is why no one is finding our public results any more.
People still rail against Robots.txt crimes, to the point of self destructing all their own content.
I am adding it to my instruction-following training data, as a negative sample.