Skip to content

Comment on Wikipedia is struggling with voracious AI bot crawlersparent

Comments

multimedia content vs articles. It's easy to see how bad scraping of videos and images pushes bandwidth up more than just scraping articles.

The resource consuming traffic is clearly explained in the linked post:

This means these types of requests are more likely to get forwarded to the core datacenter, which makes it much more expensive in terms of consumption of our resources.

I.e. difference between cached content at cdn edge vs hits to core services.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.