Comment on Ask HN: How are you dealing with scraping hits from EC2 machines?parentComments−digitalpbkOP13yMostly duplicate content & messing up my analytics (increased bounce rate, decreased time spend on page etc.)−digitalpbkOP13yBecause it seems to be from selenium (from referer), it is triggering the JS too, we are using Google Analytics.−ig113yWhy not exclude the EC2 ip range from analytics ?−dumbfounder13yWhat analytics are you using? I thought most would use javascript to avoid problems like this, and I would wager that the vast majority of bots don't bother executing javascript. You will always have legit bots hitting your site as well.−bdcravens13ySelenium executes Javascript, as do other WebKit-based scrapers, like phantom.js.
Comments
Mostly duplicate content & messing up my analytics (increased bounce rate, decreased time spend on page etc.)
Because it seems to be from selenium (from referer), it is triggering the JS too, we are using Google Analytics.
Why not exclude the EC2 ip range from analytics ?
What analytics are you using? I thought most would use javascript to avoid problems like this, and I would wager that the vast majority of bots don't bother executing javascript. You will always have legit bots hitting your site as well.
Selenium executes Javascript, as do other WebKit-based scrapers, like phantom.js.