Skip to content

Comment on Robots.txt Disallow: 20 Years of Mistakes To Avoidparent

Comments

I think conflating a publicly accessible website with a private conversation - even if in a public setting - is specious. That said, I concede the point that some websites are meant to belong to the deep web, and while I wouldn't feel guilty about archiving it for personal use anyway, I wouldn't blame the author for banning archivers.

I'd say my general rule is closer to: if you allow search engines, you should allow IA.

W.r.t. your last question, historically the solution to that problem was simple and elegant: people used pen names to write what they didn't want to bind permanently to them. This way is also safer from unauthorized archiving - not everyone is as respectful of the author's wishes as IA.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.