Skip to content

Comment on Ask HN: How are you dealing with scraping hits from EC2 machines?parent

Comments

We've blocked a few of the worst, but mostly just added servers to deal with the load.

We actually found out who one of the worst ones was and contact them. It turns out it was a major legit proxy, but they had a bug in their proxy code that caused refetching of one of our urls over and over. They were very easy to work with and they fixed the bug.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.