Comment on Anthropic is scraping websites so fast it's causing problemsparentComments−ericholscher2yYea, this happened to us, and you can see the cache profile of a crawler here:https://fosstodon.org/@readthedocs/112877477202118215A bunch of the traffic hit the origin.−sitkack2yNot trying to splain you, but if you can turn off caching for one hit wonders, everything would be greatly improved.
Comments
Yea, this happened to us, and you can see the cache profile of a crawler here:
https://fosstodon.org/@readthedocs/112877477202118215
A bunch of the traffic hit the origin.
Not trying to splain you, but if you can turn off caching for one hit wonders, everything would be greatly improved.