I am aware and have a rate limit rule for the AI bot category configured in Cloudflare which defaults to returning a 429.
But I was mostly being sarcastic. No other bots are this aggressive. I see tens of thousands of hits per day from other bots, Amazon, Apple, etc. OpenAI is the only one generating hundreds of thousands of hits per day to a small niche website. Rate limiting the AI bot category ends up impacting the better behaving bots also.
For comparison it takes Google about 45 days to make the same number of requests that OpenAI makes per day.
Comments
Yes, that's called following the http specs and return http 429, too many requests, the crawler would probably throttle in this case.
I am aware and have a rate limit rule for the AI bot category configured in Cloudflare which defaults to returning a 429.
But I was mostly being sarcastic. No other bots are this aggressive. I see tens of thousands of hits per day from other bots, Amazon, Apple, etc. OpenAI is the only one generating hundreds of thousands of hits per day to a small niche website. Rate limiting the AI bot category ends up impacting the better behaving bots also.
For comparison it takes Google about 45 days to make the same number of requests that OpenAI makes per day.