Nowadays, somebody can just ask claude to build them a scraper/bot that hooks into a proxy network and all of a sudden they can easily send 20k+ reqs/min from hundreds or thousands of IPs cycling them as they get rate limited or banned. In my work, the scrapers have gotten way more aggressive in the last 2 years or so. Frankly, I'm happy there is a solution.
There may be things to criticize Cloudflare for, but the problem of bots and scrapers destroying the open web was getting worse no matter what.
Comments
Cloudflare gains things like this:
https://blog.cloudflare.com/introducing-pay-per-crawl/
https://developers.cloudflare.com/browser-run/quick-actions/...
They create a new problem and sell the solution.
Nowadays, somebody can just ask claude to build them a scraper/bot that hooks into a proxy network and all of a sudden they can easily send 20k+ reqs/min from hundreds or thousands of IPs cycling them as they get rate limited or banned. In my work, the scrapers have gotten way more aggressive in the last 2 years or so. Frankly, I'm happy there is a solution.
There may be things to criticize Cloudflare for, but the problem of bots and scrapers destroying the open web was getting worse no matter what.
God damn it.