Depending on the exact specifics of what you mean by "canary", they have their own issues.
If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity: whatever latency you have from getting a canary from 0% to 100%, that then determines how quickly you can deploy. The "off" state of a feature flag is usually the pre-existing code, and while nothing is guaranteed to be 100% safe, it's usually a good bet. Then we can worry (post deploy) about ramping that FF from 0% to 100%. It raises the number of steps per deployment, too. (It's not just deploy, it's deploy canary, canary to 10%, canary to 20%, etc., make prod canary. I have seen each of those steps then get bogged down by people needing to have their nerves managed.)
If you mean something like "canary that feature branch" (which I have seen, more rarely) — that style has problems in "how do you keep the feature branch up to date with `main`/`prod`?" (or whatever you call your currently deployed version.) Basically, if I canary a feature branch — and lets say I need to make modifications, and those take time — and then prod is deployed to a new version later than my branch, now that new version has code that, if the user hits my canary, my canary lacks. (And yes, I've seen this in real companies, where a engineer ends up confused as to why their request is failing, b/c it is hitting a canary that is out of date / diverged from prod.)
If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity
Yes, that’s the point. I have never, not once, seen a place that emphasized high development velocity that also had anything resembling stability.
Note also that I mentioned I am an SRE / DBRE: it’s baffling and frustrating to me that companies will hire people whose job it is to create stable and reliable systems, then ignore them when they say “you’re moving too quickly.” Instead, we get treated as S-tier helpdesk.
Finally, to the mention of extra steps, it’s not that hard to automate. N ReplicaSets get rolled to the new release; if after M minutes all metrics are nominal, ramp up, else roll back and page.
I'm also an SRE¹. A lot of that does match with my experience. Esp. "I have never, not once, seen a place that emphasized high development velocity that also had anything resembling stability." — neither have I, I guess; perhaps I'm assuming that rapid iteration and stability are possible, whereas you seem to not. (And the evidence I've seen does suggest you might be right; I've certainly not found a way.)
Finally, to the mention of extra steps, it’s not that hard to automate. N ReplicaSets get rolled to the new release; if after M minutes all metrics are nominal, ramp up, else roll back and page.
I almost mentioned automated stepped rollouts. I guess it's more because there is a staged rollout, non-technical types inexorably inject bureaucracy and manual "approvals" into such things.
I think the other problem I'm faced as an SRE with automated stepped rollouts is "they take too long" and slow down velocity; which, I guess again, you don't see as problematic. I guess I don't know how to "market" that to the SWEs I work with. Even a non-staged k8s Deployment at my last job was frustrating devs due to the time it would take to roll out, even with no real steps, aside from "roll out new version", but the workload itself was so slow that that would still take significant time. (But nobody was willing to invest into "fix workload startup and shutdown times" — some huge fraction of which I think was just loading Google's Vertex library, IIRC, which is … a sign of the times.)
¹ty for your reply, too; that was … damning? skewering?, but insightful.
Comments
Depending on the exact specifics of what you mean by "canary", they have their own issues.
If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity: whatever latency you have from getting a canary from 0% to 100%, that then determines how quickly you can deploy. The "off" state of a feature flag is usually the pre-existing code, and while nothing is guaranteed to be 100% safe, it's usually a good bet. Then we can worry (post deploy) about ramping that FF from 0% to 100%. It raises the number of steps per deployment, too. (It's not just deploy, it's deploy canary, canary to 10%, canary to 20%, etc., make prod canary. I have seen each of those steps then get bogged down by people needing to have their nerves managed.)
If you mean something like "canary that feature branch" (which I have seen, more rarely) — that style has problems in "how do you keep the feature branch up to date with `main`/`prod`?" (or whatever you call your currently deployed version.) Basically, if I canary a feature branch — and lets say I need to make modifications, and those take time — and then prod is deployed to a new version later than my branch, now that new version has code that, if the user hits my canary, my canary lacks. (And yes, I've seen this in real companies, where a engineer ends up confused as to why their request is failing, b/c it is hitting a canary that is out of date / diverged from prod.)
Yes, that’s the point. I have never, not once, seen a place that emphasized high development velocity that also had anything resembling stability.
Note also that I mentioned I am an SRE / DBRE: it’s baffling and frustrating to me that companies will hire people whose job it is to create stable and reliable systems, then ignore them when they say “you’re moving too quickly.” Instead, we get treated as S-tier helpdesk.
Finally, to the mention of extra steps, it’s not that hard to automate. N ReplicaSets get rolled to the new release; if after M minutes all metrics are nominal, ramp up, else roll back and page.
I'm also an SRE¹. A lot of that does match with my experience. Esp. "I have never, not once, seen a place that emphasized high development velocity that also had anything resembling stability." — neither have I, I guess; perhaps I'm assuming that rapid iteration and stability are possible, whereas you seem to not. (And the evidence I've seen does suggest you might be right; I've certainly not found a way.)
I almost mentioned automated stepped rollouts. I guess it's more because there is a staged rollout, non-technical types inexorably inject bureaucracy and manual "approvals" into such things.
I think the other problem I'm faced as an SRE with automated stepped rollouts is "they take too long" and slow down velocity; which, I guess again, you don't see as problematic. I guess I don't know how to "market" that to the SWEs I work with. Even a non-staged k8s Deployment at my last job was frustrating devs due to the time it would take to roll out, even with no real steps, aside from "roll out new version", but the workload itself was so slow that that would still take significant time. (But nobody was willing to invest into "fix workload startup and shutdown times" — some huge fraction of which I think was just loading Google's Vertex library, IIRC, which is … a sign of the times.)
¹ty for your reply, too; that was … damning? skewering?, but insightful.