My naive take on a Cloudflare perspective wants to combine "three times is enemy action" with toddler-speed block dropping and manual clearance. What's the money reason this problem isn't handled at the ISP level?
I don't quite understand your post, but is your question why don't the ISPs of the sources of the abusive traffic sort it out?
The distributed nature of DDoS means each participating host isn't sending that much traffic, and there are often tens or hundreds of thousands of participating hosts. An ISP should verify claims of abuse before cutting off customers, and since most of the customers are presumably unaware of what their systems are doing, there will be a lot of unhappy customers and then you've got to spend a lot of customer support time on helping them clean up their systems so they can get back online.
I spent a fair amount of time sending out abuse reports for phishing / malware senders about a decade ago, and most abuse reporting addresses are a black hole. Even if you do get to someone who will do something about abuse, they won't do it quickly.
There's be a few high profile longer term DDoS attacks lately, but when I was running infra that got a lot of stuff, it was mostly people kicking the tires on DDoS as a service offerings and most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that.
Starting from there, this DDoS sounds like layer 7 DDoS which is easy to track to the immediate senders, but a ton of DDoS is volumetric stuff, often volumetric reflection attacks where the senders spoof your address. If you're getting that, best you can do is get the reflectors kicked off (or cleaned up) ... tracing back to the sending hosts means getting a reflector (and their ISPs) engaged to do a lot of labor intensive work.
All of that investigation stuff takes qualified people lots of time, that's your money reason it doesn't happen.
most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that.
Every victim (such as readthedocs), or even people sharing blocklists to avoid becoming a victim, blocking that ISP's ranges until they do clean up their network could be a convincing argument?
As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious. Such a reoccurring source of abuse needs to either clean up or find themselves a different ISP to spread harm onto the net
I get what you're saying about that this won't solve an ongoing attack right this minute, or even by next week. But if we just let it all happen then the solution is going to be either (1) we all buy equipment that can handle something like a terabit per second and arm our infrastructure to the teeth or (2) centralize all traffic through a vetting entity who decides which client gets to visit the internet today. So far we're headed towards the latter and nobody really wants that. Abuse messages will have to slowly trickle down from victims to originating ISPs to users, and if users didn't willingly sign up, then to wherever users are getting this malware (Google's app store will be a big component). Stopping this at the source seems to me a much more desirable long-term solution
www.uceprotect.net does that for email. It's a DNS blacklist that puts hole networks on it, even if "only" a individual hosts SPAMs.
It's a double-edged sword. That is how you end up with most ISPs blocking port 25 completely. If your hosting provider is on the list you are collateral damage. You yourself can do very little to remedy the situation except to beg your provider "to look into it".
What do you expect the ISPs to do in this story? We are talking about TSL connections. Block port 80 and 443 and expect the costumers to use your HTTPS-Proxy. Than they could inspect and block individual actions.
As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious.
It sure is --- but an ISP would want to observe the traffic themselves, and if it's a 90 second attack every so often, chances are they won't see it when they look. When it's volumetric reflection, you can probably tell them how to send a request and see the response, and maybe they'll contact the customer, but maybe they'll just sit on it. As a victim, the ROI for reporting just wasn't there.
I wasn't getting huge traffic flows, and I was mostly getting attacks against www, which wasn't my actual service, so making sure volumetric attacks below my interface rate were shrugged off and taking simple actions like dropping requests from http clients with user-agent Wordpress were good enough. If the volumetric attacks were much over 10G, my host would have null routed my servers, which is annoying but highly scalable --- many ISPs support a BGP blackhole community, so my host can add my attacked IP to that and their upstreams will drop inbound packets when they enter the ISPs network.
I can't find a reference now, but I've seen things that allowed for more specific blackholing, such as by source or destination port number or by protocol. If my host's ISPs are dropping all UDP and IP fragments to my IP under attack, I could keep serving my TCP traffic and ignore a huge DDoS. I wouldn't even be able to measure the size of the DDoS.
Comments
My naive take on a Cloudflare perspective wants to combine "three times is enemy action" with toddler-speed block dropping and manual clearance. What's the money reason this problem isn't handled at the ISP level?
I don't quite understand your post, but is your question why don't the ISPs of the sources of the abusive traffic sort it out?
The distributed nature of DDoS means each participating host isn't sending that much traffic, and there are often tens or hundreds of thousands of participating hosts. An ISP should verify claims of abuse before cutting off customers, and since most of the customers are presumably unaware of what their systems are doing, there will be a lot of unhappy customers and then you've got to spend a lot of customer support time on helping them clean up their systems so they can get back online.
I spent a fair amount of time sending out abuse reports for phishing / malware senders about a decade ago, and most abuse reporting addresses are a black hole. Even if you do get to someone who will do something about abuse, they won't do it quickly.
There's be a few high profile longer term DDoS attacks lately, but when I was running infra that got a lot of stuff, it was mostly people kicking the tires on DDoS as a service offerings and most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that.
Starting from there, this DDoS sounds like layer 7 DDoS which is easy to track to the immediate senders, but a ton of DDoS is volumetric stuff, often volumetric reflection attacks where the senders spoof your address. If you're getting that, best you can do is get the reflectors kicked off (or cleaned up) ... tracing back to the sending hosts means getting a reflector (and their ISPs) engaged to do a lot of labor intensive work.
All of that investigation stuff takes qualified people lots of time, that's your money reason it doesn't happen.
Every victim (such as readthedocs), or even people sharing blocklists to avoid becoming a victim, blocking that ISP's ranges until they do clean up their network could be a convincing argument?
As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious. Such a reoccurring source of abuse needs to either clean up or find themselves a different ISP to spread harm onto the net
I get what you're saying about that this won't solve an ongoing attack right this minute, or even by next week. But if we just let it all happen then the solution is going to be either (1) we all buy equipment that can handle something like a terabit per second and arm our infrastructure to the teeth or (2) centralize all traffic through a vetting entity who decides which client gets to visit the internet today. So far we're headed towards the latter and nobody really wants that. Abuse messages will have to slowly trickle down from victims to originating ISPs to users, and if users didn't willingly sign up, then to wherever users are getting this malware (Google's app store will be a big component). Stopping this at the source seems to me a much more desirable long-term solution
www.uceprotect.net does that for email. It's a DNS blacklist that puts hole networks on it, even if "only" a individual hosts SPAMs.
It's a double-edged sword. That is how you end up with most ISPs blocking port 25 completely. If your hosting provider is on the list you are collateral damage. You yourself can do very little to remedy the situation except to beg your provider "to look into it".
What do you expect the ISPs to do in this story? We are talking about TSL connections. Block port 80 and 443 and expect the costumers to use your HTTPS-Proxy. Than they could inspect and block individual actions.
It sure is --- but an ISP would want to observe the traffic themselves, and if it's a 90 second attack every so often, chances are they won't see it when they look. When it's volumetric reflection, you can probably tell them how to send a request and see the response, and maybe they'll contact the customer, but maybe they'll just sit on it. As a victim, the ROI for reporting just wasn't there.
I wasn't getting huge traffic flows, and I was mostly getting attacks against www, which wasn't my actual service, so making sure volumetric attacks below my interface rate were shrugged off and taking simple actions like dropping requests from http clients with user-agent Wordpress were good enough. If the volumetric attacks were much over 10G, my host would have null routed my servers, which is annoying but highly scalable --- many ISPs support a BGP blackhole community, so my host can add my attacked IP to that and their upstreams will drop inbound packets when they enter the ISPs network.
I can't find a reference now, but I've seen things that allowed for more specific blackholing, such as by source or destination port number or by protocol. If my host's ISPs are dropping all UDP and IP fragments to my IP under attack, I could keep serving my TCP traffic and ignore a huge DDoS. I wouldn't even be able to measure the size of the DDoS.
The report makes it pretty clear that the attack was distributed enough that profiling for blocklists was ineffective.
https://www.scispace.com/pdf/drawbridge-software-defined-ddo...
I saw this quite a few years ago. It is possibly relevant. I don't know what, if anything, came of it since then.