Comment on Experiments at AirbnbparentComments−blauwbilgorgel12yA/A testing (Null testing) or A/A/B testing gives a different effect than A/B testing.Microsoft Research suggested (http://ai.stanford.edu/~ronnyk/2009controlledExperimentsOnTh...) that you continuously run A/A tests alongside your experiments. An A/A test can:- Collect data and assess its variability for power calculations- test the experimentation system (the Null hypothesis should be rejected about 5% of the time when a 95% confidence level is used)- tell if users are split according to the planned percentages
Comments
A/A testing (Null testing) or A/A/B testing gives a different effect than A/B testing.
Microsoft Research suggested (http://ai.stanford.edu/~ronnyk/2009controlledExperimentsOnTh...) that you continuously run A/A tests alongside your experiments. An A/A test can:
- Collect data and assess its variability for power calculations
- test the experimentation system (the Null hypothesis should be rejected about 5% of the time when a 95% confidence level is used)
- tell if users are split according to the planned percentages