That tracks. I've been having issues lately with excessive traffic causing high CPU utilization for my small site. 859,530 requests yesterday reached the server (past any CDN caching) of that 160,602 of the request were from openai. That's over 18%. To further break that down:
This is after I doubled the CPUs to handle the load. Before that the site would get unresponsive and openai would stop DDoS'ing the site after 5-10 mins. Then they'd try again every single hour. It lead to my site becoming unresponsive at the top of every hour.
If only we had some way to teach computers that when you smash a site with so much traffic that it becomes unresponsive that it's bad for that site. Maybe someone will invent intelligent throttling but it'd probably require a company specializing in AI and scraping the internet...
I am aware and have a rate limit rule for the AI bot category configured in Cloudflare which defaults to returning a 429.
But I was mostly being sarcastic. No other bots are this aggressive. I see tens of thousands of hits per day from other bots, Amazon, Apple, etc. OpenAI is the only one generating hundreds of thousands of hits per day to a small niche website. Rate limiting the AI bot category ends up impacting the better behaving bots also.
For comparison it takes Google about 45 days to make the same number of requests that OpenAI makes per day.
Comments
That tracks. I've been having issues lately with excessive traffic causing high CPU utilization for my small site. 859,530 requests yesterday reached the server (past any CDN caching) of that 160,602 of the request were from openai. That's over 18%. To further break that down:
This is after I doubled the CPUs to handle the load. Before that the site would get unresponsive and openai would stop DDoS'ing the site after 5-10 mins. Then they'd try again every single hour. It lead to my site becoming unresponsive at the top of every hour.If only we had some way to teach computers that when you smash a site with so much traffic that it becomes unresponsive that it's bad for that site. Maybe someone will invent intelligent throttling but it'd probably require a company specializing in AI and scraping the internet...
Yes, that's called following the http specs and return http 429, too many requests, the crawler would probably throttle in this case.
I am aware and have a rate limit rule for the AI bot category configured in Cloudflare which defaults to returning a 429.
But I was mostly being sarcastic. No other bots are this aggressive. I see tens of thousands of hits per day from other bots, Amazon, Apple, etc. OpenAI is the only one generating hundreds of thousands of hits per day to a small niche website. Rate limiting the AI bot category ends up impacting the better behaving bots also.
For comparison it takes Google about 45 days to make the same number of requests that OpenAI makes per day.