Their web crawler / scraper "ClaudeBot" is the worst semi-legitimate crawler I've ever encountered. Extremely aggressive scraping from a ton of IP addresses with a ton of concurrent accesses, often getting lost in an infinite maze of dynamic url path combinations. Do better, please.
Comments
Their web crawler / scraper "ClaudeBot" is the worst semi-legitimate crawler I've ever encountered. Extremely aggressive scraping from a ton of IP addresses with a ton of concurrent accesses, often getting lost in an infinite maze of dynamic url path combinations. Do better, please.