"I don't think that philosophy is meant to apply to applications with hundreds of thousands of users. It's meant for the early product development phase when you have very few users. You can, however, do bucket testing, like Google and other larger players, and release updates to a tiny portion of your users to see how they react."
Totally agree with you there. That's actually something I really want to start doing after we release our new version. We've never done any type of A/B testing and it just makes sense.
On a similar note, I was talking with my friend Eric Friedman last night about this very topic. He suggested creating small buttons that people could click on, but weren't actually active and gave a "Coming Soon" message and then tracking the frequency in how often they were clicked. You could release these little buttons to 2-5% of your users and then see how they performed. And then build features around the buttons that were clicked more frequently.
Comments
"I don't think that philosophy is meant to apply to applications with hundreds of thousands of users. It's meant for the early product development phase when you have very few users. You can, however, do bucket testing, like Google and other larger players, and release updates to a tiny portion of your users to see how they react."
Totally agree with you there. That's actually something I really want to start doing after we release our new version. We've never done any type of A/B testing and it just makes sense.
On a similar note, I was talking with my friend Eric Friedman last night about this very topic. He suggested creating small buttons that people could click on, but weren't actually active and gave a "Coming Soon" message and then tracking the frequency in how often they were clicked. You could release these little buttons to 2-5% of your users and then see how they performed. And then build features around the buttons that were clicked more frequently.