The rate at which we can start tasks restricts the parallelism of our deploy. Despite us setting the MaximumPercent parameter to 200%, the ECS start-task API call has a hard limit of 10 tasks per call, and it is rate-limited. We need to call it 400 times to place all our containers in production.
From reading other comments it makes me wonder if you (Plaid) tried batching the tasks into N containers? Like if a task 50 containers, then you'd reduce the task call rate limiting by 50x...
Yeah, I mentioned this in a comment in a different thread. The duplicate containers in the task definition need to be marked as "essential" in CloudFormation to make sure our capacity doesn't degrade on container exits, and this means that one container exiting will also exit other containers in the same task. So we have a bleed-over effect where OOM in one request could cause N-1 other requests to fail.
The essential vs non-essential container designation is a little confusing. The standard use case for multi-container tasks seems to be that all containers are marked as essential, i.e. they essentially represent different services that are operating in concert on the same machine. This is definitely not the situation we're in, where each container is totally independent.
So it seems like we'd be a perfect use case for non-essential containers. However, (1) at least one container _must_ be marked as essential, and (2) non-essential containers which exit don't get restarted or replaced. This means we would still have a limited bleed-over effect (if the essential container exits, the other ones do too), and more importantly, we can't guarantee that our capacity will be robust to process exits.
non-essential containers which exit don't get restarted or replaced
That's why you have one or two essential watchdog containers which relaunch the workers. You keep a large number of them in an "idle, but hot" status to allow for bursts?
I'm a little confused by this approach; are there any non-essential containers in your suggested architecture? This sounds like the watchdog container is just a parent process that launches a bunch of subprocesses, which is definitely a workable solution, although not the one we decided to use. If there are primitives for an essential container to inspect container state and relaunch other containers in the task, that'd be great to know about.
Comments
From reading other comments it makes me wonder if you (Plaid) tried batching the tasks into N containers? Like if a task 50 containers, then you'd reduce the task call rate limiting by 50x...
Yeah, I mentioned this in a comment in a different thread. The duplicate containers in the task definition need to be marked as "essential" in CloudFormation to make sure our capacity doesn't degrade on container exits, and this means that one container exiting will also exit other containers in the same task. So we have a bleed-over effect where OOM in one request could cause N-1 other requests to fail.
The essential vs non-essential container designation is a little confusing. The standard use case for multi-container tasks seems to be that all containers are marked as essential, i.e. they essentially represent different services that are operating in concert on the same machine. This is definitely not the situation we're in, where each container is totally independent.
So it seems like we'd be a perfect use case for non-essential containers. However, (1) at least one container _must_ be marked as essential, and (2) non-essential containers which exit don't get restarted or replaced. This means we would still have a limited bleed-over effect (if the essential container exits, the other ones do too), and more importantly, we can't guarantee that our capacity will be robust to process exits.
That's why you have one or two essential watchdog containers which relaunch the workers. You keep a large number of them in an "idle, but hot" status to allow for bursts?
I'm a little confused by this approach; are there any non-essential containers in your suggested architecture? This sounds like the watchdog container is just a parent process that launches a bunch of subprocesses, which is definitely a workable solution, although not the one we decided to use. If there are primitives for an essential container to inspect container state and relaunch other containers in the task, that'd be great to know about.