Skip to content

Comment on A Curious Case of Disregarded Robots.txtparent

Comments

You are literally picking on the last org on the internet that needs to get picked on.

True but at the same time I do understand those who generally want others to abide by their Robots.txt. INAL, but ideally I would love to see the IA having the right to ignore Robots.txt (however, if some one wants to opt out of the IA they should be given the option) while others shouldn't be allowed to do so.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.