It's even worst than that. Most of the time the validity of the t-test that they are running is questionable. They are technically running an online t-test and as soon as they find significance they stop. This is fundamentally wrong, and not conclusive at all.
A few years ago when I was still working there, involved in the experimentation tooling among other things, we largely excised that particular behavior. What are you basing your assertion on?
People should be using more than one design, but that's not as commonly taught as it should be. I'm going to give a talk about that to my company's ML group in about a month, and hopefully that'll improve things where I am.
Not the OP, but I work on similar problems, albeit in a different setting (healthcare, millions+ of patients). The gist is that you have to bake experimental design into the deployment of your ML model, but in many cases a simple RCT or A/B test just won't cut it. This is largely because when you deploy a model, no matter how sophisticated or accurate, there's no guarantee that it'll actually move the needle in terms of the outcomes you care about—hence you need to run some kind of trial. At the same time, you want to maximize overall utility by not having to allocate more subjects to your control arm (or harmful, or resource-intensive and ineffective treatment arms) than you need to. This latter point is much more of a problem in medicine than it is in other settings, as you can imagine. These considerations point to adaptive designs that balance exploration/exploitation, e.g. those based on multi-armed bandits. Currently working on some cool (in my opinion) variations of MABs that incorporate domain-specific knowledge, so I could talk about this all day!
Would be really keen to hear you speak about the subject in greater detail actually. Love having to balance the practicalities of implementing a model in production and validating the outcome while not missing out on utility.
Do you know a good introduction to adaptive/sequential designs? I'm looking for something along the lines of a textbook aimed at a graduate level seminar.
say you want to try adding two features, which don't you think interact with each other, e.g. a change to the "pick this room button" and a change to the checkout flow. then, you can randomly assign users to two experiments, independently. your t-test results should then be valid if the two features are independent.
your t-test results should then be valid if the two features are independent.
Assuming that your assumption are correct on interaction effect.
You can do a hypothesis test on that assumption while including both factors (the two features). Which will clear away any doubt with a 95% confidence or hire a statistician =).
Comments
Experimental design is just a t-test? At least accord to that picture it seems that way. There are no ANOVA or interaction test?
Do websites usually just use t-test only? Like adding one feature at a time?
It's even worst than that. Most of the time the validity of the t-test that they are running is questionable. They are technically running an online t-test and as soon as they find significance they stop. This is fundamentally wrong, and not conclusive at all.
A few years ago when I was still working there, involved in the experimentation tooling among other things, we largely excised that particular behavior. What are you basing your assertion on?
Cheating on a hypothesis test... that's terrible.
People should be using more than one design, but that's not as commonly taught as it should be. I'm going to give a talk about that to my company's ML group in about a month, and hopefully that'll improve things where I am.
Not sure if you'd be willing, but I'd love a quick rundown of the high level takeaways if you'd be willing to drop them here.
Are you talking about more than one experimental design in terms of comparing the exp/control distributions or something else?
Not the OP, but I work on similar problems, albeit in a different setting (healthcare, millions+ of patients). The gist is that you have to bake experimental design into the deployment of your ML model, but in many cases a simple RCT or A/B test just won't cut it. This is largely because when you deploy a model, no matter how sophisticated or accurate, there's no guarantee that it'll actually move the needle in terms of the outcomes you care about—hence you need to run some kind of trial. At the same time, you want to maximize overall utility by not having to allocate more subjects to your control arm (or harmful, or resource-intensive and ineffective treatment arms) than you need to. This latter point is much more of a problem in medicine than it is in other settings, as you can imagine. These considerations point to adaptive designs that balance exploration/exploitation, e.g. those based on multi-armed bandits. Currently working on some cool (in my opinion) variations of MABs that incorporate domain-specific knowledge, so I could talk about this all day!
Would be really keen to hear you speak about the subject in greater detail actually. Love having to balance the practicalities of implementing a model in production and validating the outcome while not missing out on utility.
Do you know a good introduction to adaptive/sequential designs? I'm looking for something along the lines of a textbook aimed at a graduate level seminar.
take a look at this https://www.amazon.com/Multi-armed-Bandit-Allocation-Indices...
Thanks!
I'm just going to present the absolute basics of the topic. It'll be a high-level overview of something along the lines of chapters 1-4 of https://www.amazon.com/Analysis-Experiments-Chapman-Statisti....
Thanks for the reference!
say you want to try adding two features, which don't you think interact with each other, e.g. a change to the "pick this room button" and a change to the checkout flow. then, you can randomly assign users to two experiments, independently. your t-test results should then be valid if the two features are independent.
Assuming that your assumption are correct on interaction effect.
You can do a hypothesis test on that assumption while including both factors (the two features). Which will clear away any doubt with a 95% confidence or hire a statistician =).