Comment on Show HN: Crawlee – Web scraping and browser automation library for Node.jsparentComments−wumpus4yMost mentions of crawl-delay in robots.txt set a limit so slow that the website can't be fully crawled before the heat death of the Universe. That's why Google and bing etc. ignore crawl-delay.
Comments
Most mentions of crawl-delay in robots.txt set a limit so slow that the website can't be fully crawled before the heat death of the Universe. That's why Google and bing etc. ignore crawl-delay.