I wonder if this is why there's a rise in A/B testing. We normally see very minor differences, like about 54% vs 46%, but we spend a lot of engineering effort collecting, storing, comparing this data. It seems like something we should just flip a coin over, but I guess it's an anti-bikeshedding method.
The best way to prevent bikeshedding is to make it really easy to try out different colors and measure them.
As the engineer who's been in the "touch nothing" camp before, it's always because the risk/reward of making big changes to bad systems is really skewed towards things breaking and making me miserable.
I mean I feel like the process of doing A/B testing is what leads to bad systems. At the very least, you have to build path A, path B, and the null path where the A/B testing system goes down. It's a lot of work to let people change colors, far more if it's something like navigation or a new dialog. And it gets exponentially complex with more of these.
Comments
I wonder if this is why there's a rise in A/B testing. We normally see very minor differences, like about 54% vs 46%, but we spend a lot of engineering effort collecting, storing, comparing this data. It seems like something we should just flip a coin over, but I guess it's an anti-bikeshedding method.
The best way to prevent bikeshedding is to make it really easy to try out different colors and measure them.
As the engineer who's been in the "touch nothing" camp before, it's always because the risk/reward of making big changes to bad systems is really skewed towards things breaking and making me miserable.
I mean I feel like the process of doing A/B testing is what leads to bad systems. At the very least, you have to build path A, path B, and the null path where the A/B testing system goes down. It's a lot of work to let people change colors, far more if it's something like navigation or a new dialog. And it gets exponentially complex with more of these.