Skip to content

AI scraperbots keep hammering us from residential proxies

fosstodon.org
5 pointsaendruk1 comment
On HN

Comments

Been seeing this as well. This rule in Cloudflare gets almost all of them and doesn't seem to hit legitimate traffic. The bot is using old versions of browsers in the agent string. We set this with a non interactive challenge and it's been a real life saver:

(http.user_agent contains "Chrome/146." or http.user_agent contains "Chrome/133." or http.user_agent contains "Chrome/145." or http.user_agent contains "Firefox/135." or http.user_agent contains "Firefox/133." or http.user_agent contains "Chrome/131." or http.user_agent contains "Chrome/116." or http.user_agent contains "Chrome/124." or http.user_agent contains "Chrome/107." or http.user_agent contains "Chrome/104." or http.user_agent contains "Chrome/110." or http.user_agent contains "Chrome/99." or http.user_agent contains "Chrome/119." or http.user_agent contains "Chrome/136." or http.user_agent contains "Chrome/123." or http.user_agent contains "Chrome/100." or http.user_agent contains "Chrome/101." or http.user_agent contains "Chrome/65." or http.user_agent contains "Chrome/108." or http.user_agent contains "Chrome/112." or http.user_agent contains "Chrome/109." or http.user_agent contains "Chrome/103." or http.user_agent contains "Chrome/106." or http.user_agent contains "Chrome/111." or http.user_agent contains "Chrome/105." or http.user_agent contains "Chrome/117." or http.user_agent contains "Firefox/115." or http.user_agent contains "Chrome/135.") and http.referer eq "" and (not cf.client.bot) and (not http.cookie contains "xf_user")

Edit: Note, this also only happens to people who don't have a Xenforo key set, you'll need to remove that or change it to whatever cookie your software sets.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.