Unfortunately yes. They see too many requests from paperdelivery.co and block requests originating from that domain - they only want the google bot to crawl their site :) I think I have two options:
1. Respect their decision to not allow PaperDelivery to fetch that page.
2. Use a set of proxies and headless browsers or outright impersonate the Google bot to trick the news websites into allowing PaperDelivery to fetch that page.
Comments
Unfortunately yes. They see too many requests from paperdelivery.co and block requests originating from that domain - they only want the google bot to crawl their site :) I think I have two options:
1. Respect their decision to not allow PaperDelivery to fetch that page.
2. Use a set of proxies and headless browsers or outright impersonate the Google bot to trick the news websites into allowing PaperDelivery to fetch that page.