Thanks for the link. Very succinct. Coincidentally, just a week back, I ordered the book it mentions.
I completely agree with everything you mentioned (And thanks for giving in numbers too!).
Since you’ve clearly thought this through, if I can push you to the hypothetical - at what point would you want to run multiple tests in parallel? Will it be largely a function of your dev team (more productive everyone) or active users (more frequent feature build outs / cross cutting rollouts) ?
1) running multiple tests in parallel becomes important where the product/dev frequency exceeds waiting for a single test to finish, which is roughly a week for us. the business is currently relatively stable and doesn't change a lot, and I don't have big ideas that would need measurement so frequently, but if I did, I'd probably look for ways of running things in parallel. So I don't think it's a function of the dev team as much as what you reasonably want to measure. Having said that, I can imagine a larger dev team implying multiple product people, and those might want to have their own experiments/ideas measured in parallel, so there is something there definitely about the size of the dev team as well
2) if we wanted to be very scientific about the whole thing, we'd need multiple tests in parallel, particularly for novelty/primacy holdouts and to reconfirm that certain things actually hold across time. e.g. if I decided that variant A is better than variant B but our traffic changes (which it kind of it did over the last 3-4 months with AI overviews killing clicks from Google), maybe the reduced set of people we get behaves differently than the larger set we measured against a year ago. so I'd probably want to re-run critical experiments and just confirm them.
Comments
Thanks for the link. Very succinct. Coincidentally, just a week back, I ordered the book it mentions.
I completely agree with everything you mentioned (And thanks for giving in numbers too!).
Since you’ve clearly thought this through, if I can push you to the hypothetical - at what point would you want to run multiple tests in parallel? Will it be largely a function of your dev team (more productive everyone) or active users (more frequent feature build outs / cross cutting rollouts) ?
Thanks again for the note. Very helpful.
So pure theoretical thinking now, but:
1) running multiple tests in parallel becomes important where the product/dev frequency exceeds waiting for a single test to finish, which is roughly a week for us. the business is currently relatively stable and doesn't change a lot, and I don't have big ideas that would need measurement so frequently, but if I did, I'd probably look for ways of running things in parallel. So I don't think it's a function of the dev team as much as what you reasonably want to measure. Having said that, I can imagine a larger dev team implying multiple product people, and those might want to have their own experiments/ideas measured in parallel, so there is something there definitely about the size of the dev team as well
2) if we wanted to be very scientific about the whole thing, we'd need multiple tests in parallel, particularly for novelty/primacy holdouts and to reconfirm that certain things actually hold across time. e.g. if I decided that variant A is better than variant B but our traffic changes (which it kind of it did over the last 3-4 months with AI overviews killing clicks from Google), maybe the reduced set of people we get behaves differently than the larger set we measured against a year ago. so I'd probably want to re-run critical experiments and just confirm them.