Comment on Ask HN: How are you dealing with scraping hits from EC2 machines?parentComments−lifeguard13yThat is the way to block spiders that obey the standards and enough do not that robots.txt is not a solution.
Comments
That is the way to block spiders that obey the standards and enough do not that robots.txt is not a solution.