Comment on Ask HN: How are you dealing with scraping hits from EC2 machines?parentComments−dumbfounder13yCalling them "scrapers" implies they are doing something nefarious (stealing content). Robots.txt is for law abiding bots.−ig113yNot really. "Scraping" just refers to extracting data from a site using an automated method, it doesn't have any connotations about the motivation or acceptability of the process.
Comments
Calling them "scrapers" implies they are doing something nefarious (stealing content). Robots.txt is for law abiding bots.
Not really. "Scraping" just refers to extracting data from a site using an automated method, it doesn't have any connotations about the motivation or acceptability of the process.