I'm not an expert in statistics, but I'm careful about what I do, and I know the limitations of my knowledge.
However ...
I believe it is flawed methodolgy to run a test until you get a significant result. I'm pretty sure I read something lunk to from HN that discussed this at some length. It's possible - if you simply run your test until you get significance and then stop early - that you will get significance because of random fluctuations in the middle of your trial and stop early when you shouldn't.
As I recall, you should decide on the length of your trial at the beginning, then run your stats at the end.
I'll try to find the article in question, but my Google-Fu is pretty poor today for some reason.
Comments
I'm not an expert in statistics, but I'm careful about what I do, and I know the limitations of my knowledge.
However ...
I believe it is flawed methodolgy to run a test until you get a significant result. I'm pretty sure I read something lunk to from HN that discussed this at some length. It's possible - if you simply run your test until you get significance and then stop early - that you will get significance because of random fluctuations in the middle of your trial and stop early when you shouldn't.
As I recall, you should decide on the length of your trial at the beginning, then run your stats at the end.
I'll try to find the article in question, but my Google-Fu is pretty poor today for some reason.
EDIT: It's here:
http://news.ycombinator.com/item?id=1277004