Reading the title, I was expecting to see a somewhat shady article on using multiple servers to dodge API rate limits.
Instead, they've got a pretty snazzy writeup of how to effectively keep your distributed processes working together properly. I'm curious why they didn't pass a token rather than the actual Lua over the wire via Redis, but it certainly seems to work for them.
I may be missing something but I think this could be done more cleanly with an atomic counter or semaphore [1] in Redis.
1. Have a task that releases 1 resource to the semaphore
every (1.0 / cps) seconds.
2. Workers wait to acquire a resource from the semaphore
before making a call.
Due to the rate of release being fixed at 1/cps no worker can ever exceed the calls-per-second limit. Much simpler than sending Lua over the wire.
The only downside I see to this approach in the context of this post is that then you'd essentially need to dedicated daemon sending releases to the semaphore for #1.
I suppose you could use a long running celery task for that purpose. You wouldn't want to fire of (1.0 / cps) celery tasks just for releasing semaphores unless the value of cps was very small -- as it would congest your queues with a bunch of small cleanup tasks. Which aren't guaranteed to run precisely that often.
Interestingly enough, Alan Shreve gave a talk at RedisConf a couple of years back detailing something very similar used on the twilio stack to rate limit draining a queue: https://inconshreveable.com/talks.html
So they send code to Redis to make sure that they won't increment too high? I know Redis already has a ton of functions prepackaged but this seems like it would be a good one: INCRIF.
Comments
Reading the title, I was expecting to see a somewhat shady article on using multiple servers to dodge API rate limits.
Instead, they've got a pretty snazzy writeup of how to effectively keep your distributed processes working together properly. I'm curious why they didn't pass a token rather than the actual Lua over the wire via Redis, but it certainly seems to work for them.
I may be missing something but I think this could be done more cleanly with an atomic counter or semaphore [1] in Redis.
Due to the rate of release being fixed at 1/cps no worker can ever exceed the calls-per-second limit. Much simpler than sending Lua over the wire.[1] https://github.com/dv/redis-semaphore
The only downside I see to this approach in the context of this post is that then you'd essentially need to dedicated daemon sending releases to the semaphore for #1.
I suppose you could use a long running celery task for that purpose. You wouldn't want to fire of (1.0 / cps) celery tasks just for releasing semaphores unless the value of cps was very small -- as it would congest your queues with a bunch of small cleanup tasks. Which aren't guaranteed to run precisely that often.
Interestingly enough, Alan Shreve gave a talk at RedisConf a couple of years back detailing something very similar used on the twilio stack to rate limit draining a queue: https://inconshreveable.com/talks.html
So they send code to Redis to make sure that they won't increment too high? I know Redis already has a ton of functions prepackaged but this seems like it would be a good one: INCRIF.