Skip to content

Comment on Robots.txt Disallow: 20 Years of Mistakes To Avoidparent

Comments

Your last sentence is key. As I understand it, there's no real legal precedent for IA which basically copies everything out there on an opt-out basis. I personally am glad they do but one of the ways they get off with it is by treading as lightly as possible, including respecting robots.txt even retroactively.

They're also non-commercial, broad in scope, arguably serve a valuable scholarly function and have other characteristics that have kept them mostly out of legal hot water. But it's unclear to what degree they're legally different from a site that decided to create an archive of all comics, commercial and otherwise, and slap advertising up.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.