I have an (unfinished) anti-bot PHP project that does customizable automated bot tests and blocking (both via 403 forbidden, and iptables). A custom version of this script is running at one e-commerce site since 2012. The 403 page has a "ticket ID" so the (very few) real users caught can still contact the website admins to be unblocked.
The (almost feature-complete) version running at the site above was able to automatically identify and successfully block a number of bots found scraping (or, attempting to scrape) the site since 2012. Though it wasn't an easy task, it has been quite successful at blocking most of them since their second (often first) request.
In fact since the beginning, and largely upto now, most bots and botnets can still be easily spotted. Some by their user-agent, some falling into bot traps ("honeypots") which are invisible to regular users, some failing to load images (something that a regular user wouldn't do), some crawling the site at an unreasonable pace... most usually don't provide an http referer, some even appear to have a number of cookies set for your domain.
It's not always easy to spot them though - some botnets can only be undone by blocking their full IP subnet, or permanently banning their (sub)domains, like .cn, .ua or .br (if you don't have legitimate traffic from those domains).
Unfortunately, I haven't had enough spare time to complete my project (which tests all of the above conditions to spot bots, but is not ready for a release), but I could if there is some market - or I could sell the project if somebody is interested.
It's not hard to make a 70% effective bot blocker. It's extremely hard to make a 95%+ effective bot blocker without having a large number of false positives.
It is (probably) impossible to make a 99%+ effective bot blocker without having a very large number of false positives.
I've been working on this (and abusive user blocking) for a long time now and I'm roughly where you are, I have a product, it could be launched but I'm very wary of launching a half baked product that would eventually end up being disabled. I've had the help of a friendly corporate entity that gave me millions of datapoints to play with allowing me to look into their kitchen both on the support end and on the incoming traffic end as well as verified abusers.
This identified various fraud such as ad click fraud, affiliate scams, repeated harassment by the same users under different names, spam of all kinds and so on. It's been working there for more than 4 months on a fairly high level of confidence. And yet, I'm still wary of rolling it out on a larger scale.
Comments
I have an (unfinished) anti-bot PHP project that does customizable automated bot tests and blocking (both via 403 forbidden, and iptables). A custom version of this script is running at one e-commerce site since 2012. The 403 page has a "ticket ID" so the (very few) real users caught can still contact the website admins to be unblocked.
The (almost feature-complete) version running at the site above was able to automatically identify and successfully block a number of bots found scraping (or, attempting to scrape) the site since 2012. Though it wasn't an easy task, it has been quite successful at blocking most of them since their second (often first) request.
In fact since the beginning, and largely upto now, most bots and botnets can still be easily spotted. Some by their user-agent, some falling into bot traps ("honeypots") which are invisible to regular users, some failing to load images (something that a regular user wouldn't do), some crawling the site at an unreasonable pace... most usually don't provide an http referer, some even appear to have a number of cookies set for your domain.
It's not always easy to spot them though - some botnets can only be undone by blocking their full IP subnet, or permanently banning their (sub)domains, like .cn, .ua or .br (if you don't have legitimate traffic from those domains).
Unfortunately, I haven't had enough spare time to complete my project (which tests all of the above conditions to spot bots, but is not ready for a release), but I could if there is some market - or I could sell the project if somebody is interested.
It's not hard to make a 70% effective bot blocker. It's extremely hard to make a 95%+ effective bot blocker without having a large number of false positives.
It is (probably) impossible to make a 99%+ effective bot blocker without having a very large number of false positives.
I've been working on this (and abusive user blocking) for a long time now and I'm roughly where you are, I have a product, it could be launched but I'm very wary of launching a half baked product that would eventually end up being disabled. I've had the help of a friendly corporate entity that gave me millions of datapoints to play with allowing me to look into their kitchen both on the support end and on the incoming traffic end as well as verified abusers.
This identified various fraud such as ad click fraud, affiliate scams, repeated harassment by the same users under different names, spam of all kinds and so on. It's been working there for more than 4 months on a fairly high level of confidence. And yet, I'm still wary of rolling it out on a larger scale.
Your bot behavior spotting would flag me as a bot