One practical point anyone implementing bandits should be aware of is time correlation. Bandits are awesome if your data behaves the same going forward as it did before. But when you change your A/B split on Friday based on what you learned over the week, then suddenly whether someone saw A or B is correlated with being the kind of user that visits on weekends. That leads to selection bias, which is a pain in the ass and screws things up. It can be time of day, day of week, and even season (I bet all of our jobs will be a bit unusual going in to this coming week, for example). The problem's present on many time scales.
You can deal it, but everyone should be aware of this before doing the naive thing of just throwing bandits at things. Sometimes some regret is worth the plain validity of an RCT, and sometimes not.
Comments
One practical point anyone implementing bandits should be aware of is time correlation. Bandits are awesome if your data behaves the same going forward as it did before. But when you change your A/B split on Friday based on what you learned over the week, then suddenly whether someone saw A or B is correlated with being the kind of user that visits on weekends. That leads to selection bias, which is a pain in the ass and screws things up. It can be time of day, day of week, and even season (I bet all of our jobs will be a bit unusual going in to this coming week, for example). The problem's present on many time scales.
You can deal it, but everyone should be aware of this before doing the naive thing of just throwing bandits at things. Sometimes some regret is worth the plain validity of an RCT, and sometimes not.