Skip to content

Comment on Wikipedia is struggling with voracious AI bot crawlers

Comments

It's not just Wikipedia - the entire rest of the open-access web is suffering with them.

I think the most interesting thing here is that it shows that the companies doing these crawls simply don't care who they hurt, as they actively take measures to prevent their victims from stopping them by using multiple IP addresses, snowshoe crawling, evading fingerprinting, and so on.

For Wikipedia, there's a solution served up to them on a plate. But they simply can't be bothered to take it.

And this in turn shows the overall moral standards of those companies - it's the wild west out there, where the weak go to the wall, and those inflicting the damage know what they're doing, and just don't care. Sociopaths.

Truth. I have a platform with over a million photos. It costs me a lot of bandwidth.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.