This is an interesting issue and I have seen users of my app (Visual Website Optimizer) complaining that their results were statistically significant a day before but now they aren't. Justifiably, they expect significance to freeze in time once it has been achieved. However, as you say significance is also a random function and not necessarily monotonically increasing or decreasing.
The constraint here is not the math or technology rather it is users' needs. They want data, reporting and significance calculation to be done in real time. And even though we have a test duration calculator, I haven't seen any user actually making use of it. Plus many users will not even wait for statistical significance to be achieved.
Though, in VWO, we will love to wait calculating significance until end of experiment. I'm sure the users won't like it at all.
Comments
This is an interesting issue and I have seen users of my app (Visual Website Optimizer) complaining that their results were statistically significant a day before but now they aren't. Justifiably, they expect significance to freeze in time once it has been achieved. However, as you say significance is also a random function and not necessarily monotonically increasing or decreasing.
The constraint here is not the math or technology rather it is users' needs. They want data, reporting and significance calculation to be done in real time. And even though we have a test duration calculator, I haven't seen any user actually making use of it. Plus many users will not even wait for statistical significance to be achieved.
Though, in VWO, we will love to wait calculating significance until end of experiment. I'm sure the users won't like it at all.