Haven't you just moved the pressure back a step? Now you have a bunch of items retrying to get their stuff on the queue and a bunch of places to try and locate the problem, instead of an overly large queue. Is the idea that the messages aren't important enough to be placed on the queue in the first place if demand is high?
The idea is that it's not a sudden hard failure. Instead only a few things fail which helps in two ways: reducing the load by discarding some work and alerting that the system is beginning to be overloaded.
Comments
Haven't you just moved the pressure back a step? Now you have a bunch of items retrying to get their stuff on the queue and a bunch of places to try and locate the problem, instead of an overly large queue. Is the idea that the messages aren't important enough to be placed on the queue in the first place if demand is high?
The idea is that it's not a sudden hard failure. Instead only a few things fail which helps in two ways: reducing the load by discarding some work and alerting that the system is beginning to be overloaded.
this is similar to Random Early Detection [1].
[1] https://en.wikipedia.org/wiki/Random_early_detection
Yes.