Comment on Who needs to scrape millions of pages, or monitor them?parentComments−calufaOP13yCurrently it doesn't look at the robots.txt file. I had taken note of this, and will be added in future releases. Thanks for the suggestion.
Comments
Currently it doesn't look at the robots.txt file. I had taken note of this, and will be added in future releases. Thanks for the suggestion.