Some of our sites have been getting absolutely hammered by the AI bots -- so much so they are taking down the sites. Even with cloudflare protection and caching. The only thing We've been able to do so far is tell Cloudflare to block all AI bots, modify the robots.txt and even then we've had to manually identify IP addresses and bots that ignore all of the above and block them specifically or at the ASN level.
Cloudflare makes doing this kind of stuff easy but I would hate to have to do this manually on a webserver. And I don't like the idea of how much of the internet already relies on Cloudflare.
Comments
Some of our sites have been getting absolutely hammered by the AI bots -- so much so they are taking down the sites. Even with cloudflare protection and caching. The only thing We've been able to do so far is tell Cloudflare to block all AI bots, modify the robots.txt and even then we've had to manually identify IP addresses and bots that ignore all of the above and block them specifically or at the ASN level.
Cloudflare makes doing this kind of stuff easy but I would hate to have to do this manually on a webserver. And I don't like the idea of how much of the internet already relies on Cloudflare.