The first calculation that the author is setting up is a power calculation, which is a strong start. Based on your expectations about the effect size of the treatment (in this case, the difference between A and B), and your desired probability of correctly identifying a difference, you can figure out how large of a sample size you need to see an effect. (This is called Beta.)
If you're going to take several peeks as you run your trial and you want to be particularly rigorous, consider alpha spending functions. In medicine, alpha-spending functions are often used to take early looks trial results. 'Alpha' is what you use to determine which P-values you will consider significant. To oversimplify a bit, early peeks (before you've got your full sample size) have very extreme alphas. If your trial ultimately uses an alpha of 0.05, a prespecified early look may use an alpha of 0.001. (There are ways of calculating a meaningful alpha values; these are just examples drawn from a hat.)
By setting useful alphas and betas, you can benefit from true, potent treatment effects (if present) earlier than you might otherwise, without too much risk of identifying spurious associations.
Great point, do you have any recommendation for a paper on alpha spending functions? This looks interesting way to compensate for early peeking into significance.
Argh, having a hard time finding the paper I was thinking of. I thought it was in JAMA in 2009 but perhaps not. It had a lot of this information nicely graphed, but alas. I'll keep digging around and reply again if I find it.
Comments
The first calculation that the author is setting up is a power calculation, which is a strong start. Based on your expectations about the effect size of the treatment (in this case, the difference between A and B), and your desired probability of correctly identifying a difference, you can figure out how large of a sample size you need to see an effect. (This is called Beta.)
If you're going to take several peeks as you run your trial and you want to be particularly rigorous, consider alpha spending functions. In medicine, alpha-spending functions are often used to take early looks trial results. 'Alpha' is what you use to determine which P-values you will consider significant. To oversimplify a bit, early peeks (before you've got your full sample size) have very extreme alphas. If your trial ultimately uses an alpha of 0.05, a prespecified early look may use an alpha of 0.001. (There are ways of calculating a meaningful alpha values; these are just examples drawn from a hat.)
By setting useful alphas and betas, you can benefit from true, potent treatment effects (if present) earlier than you might otherwise, without too much risk of identifying spurious associations.
Great point, do you have any recommendation for a paper on alpha spending functions? This looks interesting way to compensate for early peeking into significance.
Argh, having a hard time finding the paper I was thinking of. I thought it was in JAMA in 2009 but perhaps not. It had a lot of this information nicely graphed, but alas. I'll keep digging around and reply again if I find it.