The worst bot normally doesn't identify themselves as bots.
A really useful flag would be "isAIBot" so you can tell them to f-off. A colleague returned from SRECon and had been asking around in regards to bots from AI companies and it's getting ridicules. AI companies are just hammering sites left and right, to the point where some are hitting the limits on their deals with hosting companies and transit providers.
And you can't filter them out, because their running on AWS, Azure or GCP IPs and aren't identifying themselves properly.
Comments
The worst bot normally doesn't identify themselves as bots.
A really useful flag would be "isAIBot" so you can tell them to f-off. A colleague returned from SRECon and had been asking around in regards to bots from AI companies and it's getting ridicules. AI companies are just hammering sites left and right, to the point where some are hitting the limits on their deals with hosting companies and transit providers.
And you can't filter them out, because their running on AWS, Azure or GCP IPs and aren't identifying themselves properly.
Why allow free and unrestricted connections from hosting companies in the first place?