Why are we still using p<0.05 for web A/B testing? p<0.05 made sense when each individual data point cost real money to generate: grad students interviewing participants or geologists making individual measurements. p < 0.05 was a good tradeoff between certainty and cost.
Now, in the world of the web where measurement has an upfront cost but 0 incremental cost, why not move to p < 0.001 or p < 0.0001? Sure, you need to increase the magnitude of data you're gathering by 2 or 3 but that's so much easier than delving into the epistemological complexities of p < 0.05
Comments
Why are we still using p<0.05 for web A/B testing? p<0.05 made sense when each individual data point cost real money to generate: grad students interviewing participants or geologists making individual measurements. p < 0.05 was a good tradeoff between certainty and cost.
Now, in the world of the web where measurement has an upfront cost but 0 incremental cost, why not move to p < 0.001 or p < 0.0001? Sure, you need to increase the magnitude of data you're gathering by 2 or 3 but that's so much easier than delving into the epistemological complexities of p < 0.05