While I understand the merit of what this article is saying I really want to caution always requiring a strict high confidence level when making a decision in start-ups. Requiring a strict confidence level does make sense for a company like Zynga who has nearly a limitless supply of users to run tests on, but for a start-up the value of being able to make a decision quickly often outweighs the value of being '95% confident'. Let's not forget all this wasted time worrying about the details of all of this math.
In my opinion, peak early and often and when your gut tells you something is true, it probablly is.
It is actually a mathematical fact if at any point in your A/B tests A is bigger than B, based on that data there is at least a 50% prob that asymptotically A is bigger than B.
This article neither advocates a high nor low confidence level. The point is that your confidence level is meaningless if you don't fix the sample size.
"...when your gut tells you something is true, it probablly is."
If you run a data-driven business with a philosophy like that, you've rewound management science to about 1700 AD. Human "guts" aren't evolved for evaluating UX effectiveness from sparse data.
Comments
While I understand the merit of what this article is saying I really want to caution always requiring a strict high confidence level when making a decision in start-ups. Requiring a strict confidence level does make sense for a company like Zynga who has nearly a limitless supply of users to run tests on, but for a start-up the value of being able to make a decision quickly often outweighs the value of being '95% confident'. Let's not forget all this wasted time worrying about the details of all of this math.
In my opinion, peak early and often and when your gut tells you something is true, it probablly is.
It is actually a mathematical fact if at any point in your A/B tests A is bigger than B, based on that data there is at least a 50% prob that asymptotically A is bigger than B.
This article neither advocates a high nor low confidence level. The point is that your confidence level is meaningless if you don't fix the sample size.
"...when your gut tells you something is true, it probablly is."
If you run a data-driven business with a philosophy like that, you've rewound management science to about 1700 AD. Human "guts" aren't evolved for evaluating UX effectiveness from sparse data.