Which should be a pretty big wake up call to anyone not doing the math. 1 in 20 events happen all the time, yet naively reading the results makes it look like one is almost certainly better than the other.
Agreed, this is such a common problem. Another really common manifestation is when you A/B test while looking at multiple variables. If you observe the results of 20 variables, you should expect one of them to have 95% confidence!
Comments
Which should be a pretty big wake up call to anyone not doing the math. 1 in 20 events happen all the time, yet naively reading the results makes it look like one is almost certainly better than the other.
Agreed, this is such a common problem. Another really common manifestation is when you A/B test while looking at multiple variables. If you observe the results of 20 variables, you should expect one of them to have 95% confidence!