If I'm reading this right, then this approach takes away any real safety in terms of deployment. There would be no easy rollback mechanism, and no real assurances that the new code version will actually run.
I understand that the main goal here seemed to be avoiding time spent in ECS rollouts, but this solution seems to be sacrificing many of the guarantees that the rollout process is designed to provide.
The root problem is explicitly called out (slow ECS deployments), and is tied to rate limiting of the ECS `start-task` API call. The post mentions the hard cap on the number of tasks per call, but I'm curious if the actual _rate limit_ could have been increased on the AWS side. Ie, 400 calls would still be needed, but they could be pushed through much faster.
Hey, great questions, some of the same questions we (the SRE/Platform Team at Plaid) had.
Plaid's rollback job still works the same way for the service using this new deployment, so thankfully, nothing new for engineers at Plaid to learn there.
We also have metrics in Prometheus to indicate which versions of code are running so we can easily verify what is deployed.
WRT the rate limit, we have a great relationship with AWS and Plaid pushes hard on limits which most often AWS is happy to increase for us, but this was a hard limit that could not be raised at the time, but I'm sure they are working on it.
You spin up 1.0.1, leaving traffic pointed at 1.0.0
What mechanism actually shifts the traffic from the 1.0.0 instances to the 1.0.1 instances, waiting for all traffic to stop on 1.0.0 before bringing the instances down without causing abrupt connection hangups?
Comments
If I'm reading this right, then this approach takes away any real safety in terms of deployment. There would be no easy rollback mechanism, and no real assurances that the new code version will actually run.
I understand that the main goal here seemed to be avoiding time spent in ECS rollouts, but this solution seems to be sacrificing many of the guarantees that the rollout process is designed to provide.
The root problem is explicitly called out (slow ECS deployments), and is tied to rate limiting of the ECS `start-task` API call. The post mentions the hard cap on the number of tasks per call, but I'm curious if the actual _rate limit_ could have been increased on the AWS side. Ie, 400 calls would still be needed, but they could be pushed through much faster.
Hey, great questions, some of the same questions we (the SRE/Platform Team at Plaid) had.
Plaid's rollback job still works the same way for the service using this new deployment, so thankfully, nothing new for engineers at Plaid to learn there.
We also have metrics in Prometheus to indicate which versions of code are running so we can easily verify what is deployed.
WRT the rate limit, we have a great relationship with AWS and Plaid pushes hard on limits which most often AWS is happy to increase for us, but this was a hard limit that could not be raised at the time, but I'm sure they are working on it.
State:
Version 1.0.0 in prod, serving requests
You want to deploy 1.0.1
You spin up 1.0.1, leaving traffic pointed at 1.0.0
What mechanism actually shifts the traffic from the 1.0.0 instances to the 1.0.1 instances, waiting for all traffic to stop on 1.0.0 before bringing the instances down without causing abrupt connection hangups?