That's why I said "retry/drop". Handling back-pressure means you must propagate it backwards to a point in your program where there is enough context to enable decision to be made on how to handle the failure. Also, a retry can wait some amount of time until the system is less busy (and this can be enforced by some other part if needed). Further, you may have many callers and one central thing processing requests so propagating back-pressure to callers also allows you to push the problem back to a point where there are potentially more resources. In a silly example, if someone spamming your queue starts to experience a sluggish UI or run out of RAM because back-pressure has forced the queue to grow on their machine an not your infrastructure, perhaps they'll get the message and cool it (ideally the program handles this gracefully too and tells the user a maximum has been hit, but even without that back-pressure is useful).
That's the best scenario; regardless of whether the event reaches the destination or not, in most cases (except continuous monitoring of some parameter which admits losses) the source still will keep the information or the way to generate that information, so the amount of state that the queue at the source requires is still being used. But moreover, the source can throttle the generation of events, or it can just decide that that event is stale and can be dropped, or can pack that event with the next events, or can signal other processes about what is happening, or it can choose other path, or maybe it has some out-of-band signaling capability, or....
Even worse, it multiplies the amount of requests hitting the overloaded processor. You can create a "metastable failure" situation that will prevent an overloaded service from ever recovering.
It helps if your client retries have backoff (but that's not always possible), but you do need to have a way to drop requests much faster than you typically process them and it's nice to have a sensible way to drop requests that are unlikely to be successful. Sometimes you can drop based on time in queue, sometimes you can drop based on percentage, sometimes you can track backend success and drop requests to a failing backend, sometimes you have to just flush the whole queue periodically if overloaded. Facebook has published something about switching from a queue to a stack during heavy load, which can work, although that stack needs bounds, too.
Even worse, it multiplies the amount of requests hitting the overloaded processor.
How is that? The queue goes between the "client" and the "server". The client thread does the work of trying to enqueue something, while the server continues to churn through existing requests completely independently.
"backpressure" is an unfortunate term because it's not something the queue does.
"back-off" would probably be a better term, since it's the behavior of the producer. The producer has to be aware of it.
"backpressure" quite literally refers to a queue blocking when full. Exerting backpressure on a system is absolutely something a queue does. Backing off or "loadshedding" is something the producer does in response to backpressure from a queue.
The point I am trying to make is that backpressure is a contract between producer and consumer. It requires a shared understanding by the both sides.
When you put a queue in your HTTP server, you are implementing both the producer (the code accepting connections) and the consumer (the code handling requests).
TCP itself also works on a shared understanding that the producer will not send more data until it was ACK-ed. If a sender kept sending packets as fast as it can, that would be a DDOS attack, and you can't stop a network level DDOS attack by adding a queue, you can only be fast enough at discarding malicious packets. So back-pressure absolutely requires the producer to behave.
You don't even need a queue for back-pressure. Java reactive streams create back-pressure without queues by making the consumers "request" data. In fact TCP would work "fine" without a queue (i.e. with a window size of 1) it would just be slow. In a sense queues are actually used to avoid back-pressure, not to create it.
Backpressure is not a contract. A protocol is a contract and there are plenty of examples of algorithms deployed that account for the existence of backpressure. But you do not need the consumer to do anything special for a system to have backpressure.
I still don't understand your point about queues not creating backpressure. If we're getting abstract about it, everything has a queue size of 1. In the article and in this thread we're talking about infinite vs bounded queues. An ideal infinite queue has no backpressure because anything can be immediately queued. However, it comes at the unfortunate expense of infinite processing time. The difference between an infinite queue and a bounded queue is that a bounded queue, when it fills up, propagates backpressure. If you choose a queue size of 1 then whatever, it doesn't change anything conseptually. You're just trying to say a queue size of 1 is not a queue and you add a queue when you go from size 1 to > 1. And now we're just arguing semantics.
Making a bounded queue bigger adds slack and relieves pressure caused by unpredictable producers. But it does not change the maximum throughput of your system. The only thing that can do that is adding more consumers or making the existing ones faster (either by improving their speed or allowing more sophistication in the handling of requests in the queue, e.g. batch processing). If you system is already at peak load, adding a queue does not relieve pressure.
I agree with everything you just said, it's just that in my (maybe twisted) mind the principle of back-pressure can be implemented in a number of ways and adding a blocking queue in front of a system that is falling over is just one of those ways. But yeah, I guess at some level everything is a queue (or has a queue in it) so you win.
Yes but that queue is bounded too. And if you follow the systems backward eventually you end up with a real person. That is then forced to prioritize away less important tasks.
Comments
Isn’t a retry just extending the queue to the caller?
That's why I said "retry/drop". Handling back-pressure means you must propagate it backwards to a point in your program where there is enough context to enable decision to be made on how to handle the failure. Also, a retry can wait some amount of time until the system is less busy (and this can be enforced by some other part if needed). Further, you may have many callers and one central thing processing requests so propagating back-pressure to callers also allows you to push the problem back to a point where there are potentially more resources. In a silly example, if someone spamming your queue starts to experience a sluggish UI or run out of RAM because back-pressure has forced the queue to grow on their machine an not your infrastructure, perhaps they'll get the message and cool it (ideally the program handles this gracefully too and tells the user a maximum has been hit, but even without that back-pressure is useful).
So, yes. But that's the point.
That's the best scenario; regardless of whether the event reaches the destination or not, in most cases (except continuous monitoring of some parameter which admits losses) the source still will keep the information or the way to generate that information, so the amount of state that the queue at the source requires is still being used. But moreover, the source can throttle the generation of events, or it can just decide that that event is stale and can be dropped, or can pack that event with the next events, or can signal other processes about what is happening, or it can choose other path, or maybe it has some out-of-band signaling capability, or....
Even worse, it multiplies the amount of requests hitting the overloaded processor. You can create a "metastable failure" situation that will prevent an overloaded service from ever recovering.
It helps if your client retries have backoff (but that's not always possible), but you do need to have a way to drop requests much faster than you typically process them and it's nice to have a sensible way to drop requests that are unlikely to be successful. Sometimes you can drop based on time in queue, sometimes you can drop based on percentage, sometimes you can track backend success and drop requests to a failing backend, sometimes you have to just flush the whole queue periodically if overloaded. Facebook has published something about switching from a queue to a stack during heavy load, which can work, although that stack needs bounds, too.
How is that? The queue goes between the "client" and the "server". The client thread does the work of trying to enqueue something, while the server continues to churn through existing requests completely independently.
"backpressure" is an unfortunate term because it's not something the queue does. "back-off" would probably be a better term, since it's the behavior of the producer. The producer has to be aware of it.
"backpressure" quite literally refers to a queue blocking when full. Exerting backpressure on a system is absolutely something a queue does. Backing off or "loadshedding" is something the producer does in response to backpressure from a queue.
The point I am trying to make is that backpressure is a contract between producer and consumer. It requires a shared understanding by the both sides.
When you put a queue in your HTTP server, you are implementing both the producer (the code accepting connections) and the consumer (the code handling requests).
TCP itself also works on a shared understanding that the producer will not send more data until it was ACK-ed. If a sender kept sending packets as fast as it can, that would be a DDOS attack, and you can't stop a network level DDOS attack by adding a queue, you can only be fast enough at discarding malicious packets. So back-pressure absolutely requires the producer to behave.
You don't even need a queue for back-pressure. Java reactive streams create back-pressure without queues by making the consumers "request" data. In fact TCP would work "fine" without a queue (i.e. with a window size of 1) it would just be slow. In a sense queues are actually used to avoid back-pressure, not to create it.
Backpressure is not a contract. A protocol is a contract and there are plenty of examples of algorithms deployed that account for the existence of backpressure. But you do not need the consumer to do anything special for a system to have backpressure.
I still don't understand your point about queues not creating backpressure. If we're getting abstract about it, everything has a queue size of 1. In the article and in this thread we're talking about infinite vs bounded queues. An ideal infinite queue has no backpressure because anything can be immediately queued. However, it comes at the unfortunate expense of infinite processing time. The difference between an infinite queue and a bounded queue is that a bounded queue, when it fills up, propagates backpressure. If you choose a queue size of 1 then whatever, it doesn't change anything conseptually. You're just trying to say a queue size of 1 is not a queue and you add a queue when you go from size 1 to > 1. And now we're just arguing semantics.
Making a bounded queue bigger adds slack and relieves pressure caused by unpredictable producers. But it does not change the maximum throughput of your system. The only thing that can do that is adding more consumers or making the existing ones faster (either by improving their speed or allowing more sophistication in the handling of requests in the queue, e.g. batch processing). If you system is already at peak load, adding a queue does not relieve pressure.
I agree with everything you just said, it's just that in my (maybe twisted) mind the principle of back-pressure can be implemented in a number of ways and adding a blocking queue in front of a system that is falling over is just one of those ways. But yeah, I guess at some level everything is a queue (or has a queue in it) so you win.
I mean it’s not about winning I think we’re mostly just saying the same thing differently at this point XD
Yes but that queue is bounded too. And if you follow the systems backward eventually you end up with a real person. That is then forced to prioritize away less important tasks.