Their rollout strategy was bad. They tested it on 1% of beta users for a month. They should have ramped it to 100% in beta before thinking about prod. Maybe 1% for a week, 25% for a week, 75% for a week then 100% for a week.
Then think about enabling it in prod. Testing on a subset of the subset of people that run beta is not enough to validate the functionality.
Even then, they probably should have done a ramp in prod as well. There are certain configurations that seem unlikely to be tested in beta. VDI is one that comes to mind. Headless operations also seem less likely to be running the beta build.
Comments
Their rollout strategy was bad. They tested it on 1% of beta users for a month. They should have ramped it to 100% in beta before thinking about prod. Maybe 1% for a week, 25% for a week, 75% for a week then 100% for a week.
Then think about enabling it in prod. Testing on a subset of the subset of people that run beta is not enough to validate the functionality.
Even then, they probably should have done a ramp in prod as well. There are certain configurations that seem unlikely to be tested in beta. VDI is one that comes to mind. Headless operations also seem less likely to be running the beta build.