I can tell you about my experience with blocking traffic from scalpers bots that were very active during pandemic.
All requests produced by those bots were valid ones, nothing that could be flagged by tools like fail2ban etc (my assumption is that it would be the same for financial systems).
Any blocking or rate limiting by IP is useless, we saw about 2-3 requests per minute per IP, and those actors had access to ridiculous number of large CIDRs, blocking any IP caused it instantly replace it with another.
blocking by AS number was also mixed bag, as this list growed up really quickly, most of that were registered to suspicious looking Gmail addresses. (I feel that such activity might own significant percentage of total ipv4 space)
This was basically cat and mouse game of finding some specific characteristic in requests that matches all that traffic and filtering it, but the other side would adapt next day or on Sunday.
aggregated amount of traffic was in range of 2-20k r/s to basically heaviest endpoint in the shop, with was the main reason we needed to block that traffic (it generated 20-40x load of organic traffic)
cloudflare was also not really successful with default configuration, we had to basically challenge everyone by default with whitelist of most common regions from where we expected customers.
So best solution is to track everyone and calculate long term reputation.
TBC: I wasn't saying that F2B is a silver bullet. Not at all.
But that protection depends on the use case. And that in many of my use-cases, a simple f2b with a large hardcoded list of URL paths I guarantee to never have, will drop bot-traffic with 90% or more. The last 10% then split into "hits because the IP is new" and "other, more sophisticated bots".
Bots, in those cases are mostly just stupid worms, just trying out known WP exploits, default passwords on often used tools (nextcloud, phpmyadmin, etc) and so on.
I've done something similar with a large list of known harvest/scraper bots, based on their user-agent (the nice ones), or their movements. Nothing complex, just things like "/hidden-page.html that's linked, but hidden with css/js.
And with spam bots, where certain post-requests can only come from repeatedly submitting the contact form.
This, obviously isn't going to give any protection against targeted attacks. Nor will it protect against more sophisticated bots. But in some -in my case, most- use-cases, it's enough to drop bot-traffic significantly.
Blocking scalper bot traffic by any means, be it by source or certified identification seems a lost cause, i.e. not possible because it can always be circumvented. Why did you not have that filter at point of sale instead? I'm sure there are reasons, but to have a battery of captchas and a limit on purchases per credit card seems on the surface much more sturdy. And it doesn't require that everyone browsing the internet announce their full name and residential address in order to satisfy the requirements of a social score ...
The product they tried to buy what not in stock anyways, but their strategy was to constantly try anyways, so in case it would become in stock they would be the first to get it.
It was all for guest checkout, so no address yet to validate nor credit card.
Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements.
As stated before the main reason we needed to block it was volume of the traffic, you migh imagine identical scenario for dealing with DDoS attack.
Disabling guest checkout would have been my weapon of choice or at least requiring the user to enter an email address to so that they are notified when the product becomes available.
> Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements
A time sensitive hash validating each request makes it a bit harder for them without significant extra work on your part. Address sensitive is much more effective but can result in issues for users that switch between networks (using your site on the move and passing between workers networks, for instance).
Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements.
That doesn't compute... Captcha is almost always used in such setups.
It also looks like you could just offer an API endpoint which would return if the article is in stock or not, or even provide a webhook. Why fight them? Just make the resource usage lighter.
I'm curious now though what the articles were, if you are at liberty to share?
We had captcha, but it was at later stage of the checkout process.
This API endpoint needed to work from cached pages, so it could not contain any dynamic state in request.
Some bots checked product page where we had info if product is in stock (although they tried heavenly to bypass any caches by putting garbage in URL). This kind of bots also scaled instantly to thousands checkout requests when product become available with gave no time for auto scaling to react (this was another challenge here)
This was easy to mitigate so it didn't generate almost any load on the system.
I believe we had email notification available, but it could be too high latency way for them.
I'm not sure how much I can share about articles here, but I can say that those were fairly expensive (and limited series) wardrobe products.
Hm, is probably too late, but you could have implemented in your API calls some kind of proof of work. Something that's not too onerous for a casual user but it is hard for someone trying multiple requests.
This was actually one of my ideas how to solve it, observed behaviour strongly suggested that all those thousands of IP addresses where used by single server. Even small PoW with this volume should heavly influence their capacity.
But we decided that we did not want to affect performance of mobile users.
We later learned that such strategy is also used by cloudflare js check
Why fight them? Just make the resource usage lighter.
Because you presumably want real, returning customers, and that means those customers need to get a chance at buying those products, instead of them being scooped up by a scalper the millisecond they appear on the website.
No matter what the price, they would still have the "As stated before the main reason we needed to block it was volume of the traffic" problem that was stated above, for a popular item. In fact increasing the base price might attract even more scalpers and such.
The best solution is to put everyone in a little cage and point and keep a permanent record of everything they do. This doesn't mean it's a desirable solution.
Comments
I can tell you about my experience with blocking traffic from scalpers bots that were very active during pandemic.
All requests produced by those bots were valid ones, nothing that could be flagged by tools like fail2ban etc (my assumption is that it would be the same for financial systems).
Any blocking or rate limiting by IP is useless, we saw about 2-3 requests per minute per IP, and those actors had access to ridiculous number of large CIDRs, blocking any IP caused it instantly replace it with another.
blocking by AS number was also mixed bag, as this list growed up really quickly, most of that were registered to suspicious looking Gmail addresses. (I feel that such activity might own significant percentage of total ipv4 space)
This was basically cat and mouse game of finding some specific characteristic in requests that matches all that traffic and filtering it, but the other side would adapt next day or on Sunday.
aggregated amount of traffic was in range of 2-20k r/s to basically heaviest endpoint in the shop, with was the main reason we needed to block that traffic (it generated 20-40x load of organic traffic)
cloudflare was also not really successful with default configuration, we had to basically challenge everyone by default with whitelist of most common regions from where we expected customers.
So best solution is to track everyone and calculate long term reputation.
TBC: I wasn't saying that F2B is a silver bullet. Not at all.
But that protection depends on the use case. And that in many of my use-cases, a simple f2b with a large hardcoded list of URL paths I guarantee to never have, will drop bot-traffic with 90% or more. The last 10% then split into "hits because the IP is new" and "other, more sophisticated bots". Bots, in those cases are mostly just stupid worms, just trying out known WP exploits, default passwords on often used tools (nextcloud, phpmyadmin, etc) and so on.
I've done something similar with a large list of known harvest/scraper bots, based on their user-agent (the nice ones), or their movements. Nothing complex, just things like "/hidden-page.html that's linked, but hidden with css/js.
And with spam bots, where certain post-requests can only come from repeatedly submitting the contact form.
This, obviously isn't going to give any protection against targeted attacks. Nor will it protect against more sophisticated bots. But in some -in my case, most- use-cases, it's enough to drop bot-traffic significantly.
I've learned that Akamai has a service that deals with this specific problem, maybe this might interest you as well: https://www.akamai.com/products/content-protector
Blocking scalper bot traffic by any means, be it by source or certified identification seems a lost cause, i.e. not possible because it can always be circumvented. Why did you not have that filter at point of sale instead? I'm sure there are reasons, but to have a battery of captchas and a limit on purchases per credit card seems on the surface much more sturdy. And it doesn't require that everyone browsing the internet announce their full name and residential address in order to satisfy the requirements of a social score ...
The product they tried to buy what not in stock anyways, but their strategy was to constantly try anyways, so in case it would become in stock they would be the first to get it. It was all for guest checkout, so no address yet to validate nor credit card. Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements.
As stated before the main reason we needed to block it was volume of the traffic, you migh imagine identical scenario for dealing with DDoS attack.
Disabling guest checkout would have been my weapon of choice or at least requiring the user to enter an email address to so that they are notified when the product becomes available.
> Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements
A time sensitive hash validating each request makes it a bit harder for them without significant extra work on your part. Address sensitive is much more effective but can result in issues for users that switch between networks (using your site on the move and passing between workers networks, for instance).
That doesn't compute... Captcha is almost always used in such setups.
It also looks like you could just offer an API endpoint which would return if the article is in stock or not, or even provide a webhook. Why fight them? Just make the resource usage lighter.
I'm curious now though what the articles were, if you are at liberty to share?
We had captcha, but it was at later stage of the checkout process. This API endpoint needed to work from cached pages, so it could not contain any dynamic state in request.
Some bots checked product page where we had info if product is in stock (although they tried heavenly to bypass any caches by putting garbage in URL). This kind of bots also scaled instantly to thousands checkout requests when product become available with gave no time for auto scaling to react (this was another challenge here)
This was easy to mitigate so it didn't generate almost any load on the system.
I believe we had email notification available, but it could be too high latency way for them.
I'm not sure how much I can share about articles here, but I can say that those were fairly expensive (and limited series) wardrobe products.
Hm, is probably too late, but you could have implemented in your API calls some kind of proof of work. Something that's not too onerous for a casual user but it is hard for someone trying multiple requests.
This was actually one of my ideas how to solve it, observed behaviour strongly suggested that all those thousands of IP addresses where used by single server. Even small PoW with this volume should heavly influence their capacity. But we decided that we did not want to affect performance of mobile users. We later learned that such strategy is also used by cloudflare js check
Because you presumably want real, returning customers, and that means those customers need to get a chance at buying those products, instead of them being scooped up by a scalper the millisecond they appear on the website.
maybe offer the item for preorder ?
Sounds like a dream having customers scooping up your products the millisecond the appear on the website. They should increase their prices.
No matter what the price, they would still have the "As stated before the main reason we needed to block it was volume of the traffic" problem that was stated above, for a popular item. In fact increasing the base price might attract even more scalpers and such.
I remember people doing this with PS5 when they were in short supply after release.
The best solution is to put everyone in a little cage and point and keep a permanent record of everything they do. This doesn't mean it's a desirable solution.
Require a verified account to buy high demand items.
why not charge people? This is the only solution I can think of.