I'll give an example of a situation where I recently had to write a throttle function that works in the same way as this library.
We have a TCP connection down which we need to send (soft) real-time updates about a large data set. However, the thread sending out the updates has a fixed sized buffer per client that's shared between every part of the system communicating with that particular client. If the buffer gets full, it disconnects the client (incorrectly reasoning the fault was the client was requesting too much data).
In certain, not particularly uncommon circumstances, it was possible to end up needing to send out a large number of updates which would blow the buffer and disconnect the client. We considered various fixes that would give feedback about the size of the buffer, but the only feedback that would reliably not blow the buffer would be the block the producer when the buffer got full. This obviously has a huge rippling effect throughout the rest of the system, some of which have firmer real-time requirements, and was too big a task to fix at that time.
Looking at the updates, we observed there were two types of data patterns: frequent short bursts and these (comparatively) rare long sequence of updates. By rate limiting them to a maximum of n messages in any given window of a second the frequent short bursts would get sent immediately and unimpeded, while the long sequences would be broken up, giving the previous messages time to be dequeued from the buffer and sent down the TCP connection before the next batch arrived.
It's crude, but so far it's worked and given us time to work out a better system-wide solution to the problem.
Comments
I'll give an example of a situation where I recently had to write a throttle function that works in the same way as this library.
We have a TCP connection down which we need to send (soft) real-time updates about a large data set. However, the thread sending out the updates has a fixed sized buffer per client that's shared between every part of the system communicating with that particular client. If the buffer gets full, it disconnects the client (incorrectly reasoning the fault was the client was requesting too much data).
In certain, not particularly uncommon circumstances, it was possible to end up needing to send out a large number of updates which would blow the buffer and disconnect the client. We considered various fixes that would give feedback about the size of the buffer, but the only feedback that would reliably not blow the buffer would be the block the producer when the buffer got full. This obviously has a huge rippling effect throughout the rest of the system, some of which have firmer real-time requirements, and was too big a task to fix at that time.
Looking at the updates, we observed there were two types of data patterns: frequent short bursts and these (comparatively) rare long sequence of updates. By rate limiting them to a maximum of n messages in any given window of a second the frequent short bursts would get sent immediately and unimpeded, while the long sequences would be broken up, giving the previous messages time to be dequeued from the buffer and sent down the TCP connection before the next batch arrived.
It's crude, but so far it's worked and given us time to work out a better system-wide solution to the problem.