I've been introducing feature flags into our component at work.
The reason why rollbacks isn't sufficient for us is that our service is semi-stateful (postgres connections are stateful, we proxy those connections). Because of this, we always keep around old pods for 5 days to let connections drain.
A deploy+rollback ends up with 3x the pods lying around, and if we deploy a fix patch that's now 4x - and if we don't deploy the fix we have 2 weeks of changes piled up for the next release.
Because of this, we instead use the feature flag. We can toggle it on and off very quickly for risky changes, and it makes no changes to pod counts
The design is postgres. It's our customers that do stateful things on those connections. Unfortunately it's tools like pgdump/pgrestore that are not tolerant to connection interruption.
Node failure is indeed a problem. Not much we can do there. Fortunately it's rare enough historically.
We are working to split the code so that the process that holds tcp connections isn't the process that handles the logic though, so it's easier to update inplace
What does it mean to do “stateful things on a connection”? Feel free to point me to literature if you’d be repeating something already written or well known.
Comments
I've been introducing feature flags into our component at work.
The reason why rollbacks isn't sufficient for us is that our service is semi-stateful (postgres connections are stateful, we proxy those connections). Because of this, we always keep around old pods for 5 days to let connections drain.
A deploy+rollback ends up with 3x the pods lying around, and if we deploy a fix patch that's now 4x - and if we don't deploy the fix we have 2 weeks of changes piled up for the next release.
Because of this, we instead use the feature flag. We can toggle it on and off very quickly for risky changes, and it makes no changes to pod counts
Out of curiosity, why can’t you change the behavior that prevents you from doing rollbacks?
This design seems rather brittle. What happens if a node spontaneously fails, or you need to move to another AZ? (Assuming you’re in the cloud.)
The design is postgres. It's our customers that do stateful things on those connections. Unfortunately it's tools like pgdump/pgrestore that are not tolerant to connection interruption.
Node failure is indeed a problem. Not much we can do there. Fortunately it's rare enough historically.
We are working to split the code so that the process that holds tcp connections isn't the process that handles the logic though, so it's easier to update inplace
What does it mean to do “stateful things on a connection”? Feel free to point me to literature if you’d be repeating something already written or well known.