Skip to content

Comment on A Curious Case of Disregarded Robots.txtparent

Comments

How is this any different from a human doing the same? The internet is meant to be open, it's free information after all. If you don't like it put your stuff behind a login

I see it as a subtle difference of automation and scale, plus the fact that the Internet Archive is not just saving these copies, but also making them available.

Imagine standing on the public road and taking a picture of your neighbor's home (or face) for your own use. Is that the same as a large company taking pictures of all homes (or faces) of the world, and making them available to the entire world, forever?

Tons of companies take satellite views of the entire earth and they're freely available online.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.