When we start processing a request, we take the JS Promise that represents the result of that operation and put it into an array in our internal "server state" object. When the server gets a shutdown request, it basically awaits Promise.all(this.inFlightRequests) before exiting. Depending on the LB/routing layer on top of your service, you might need to do this in a loop in case work comes in after you do your first await. In other languages you can join a thread pool, wait on a wait group, or use whatever other synchronization tools are provided.
Note that there's an ECS_CONTAINER_STOP_TIMEOUT parameter in ECS that sets a hard upper limit on how long containers have to exit before getting SIGKILLed, and it defaults to 30 seconds. If you want to allow requests to drain for longer than that, you'll need to update that parameter.
(For services with fast request processing time, the draining process is often a lot simpler. You can just route traffic away from the server, then wait a short period of time before telling the process to exit. It's not quite as precise, but works well for many use cases and requires no bookkeeping.)
Comments
When we start processing a request, we take the JS Promise that represents the result of that operation and put it into an array in our internal "server state" object. When the server gets a shutdown request, it basically awaits Promise.all(this.inFlightRequests) before exiting. Depending on the LB/routing layer on top of your service, you might need to do this in a loop in case work comes in after you do your first await. In other languages you can join a thread pool, wait on a wait group, or use whatever other synchronization tools are provided.
Note that there's an ECS_CONTAINER_STOP_TIMEOUT parameter in ECS that sets a hard upper limit on how long containers have to exit before getting SIGKILLed, and it defaults to 30 seconds. If you want to allow requests to drain for longer than that, you'll need to update that parameter.
(For services with fast request processing time, the draining process is often a lot simpler. You can just route traffic away from the server, then wait a short period of time before telling the process to exit. It's not quite as precise, but works well for many use cases and requires no bookkeeping.)