Comment on Ask HN: How are you dealing with scraping hits from EC2 machines?parentComments−bdcravens13yYou'd end up blocking all traffic then. When was was the last time you pulled robots.txt?
Comments
You'd end up blocking all traffic then. When was was the last time you pulled robots.txt?