I'm guessing you would have have an idea of what the right answer was going to be before you ran the analysis. How often was your guess wrong? How often did the data tell you to do something that you were uncomfortable with? Did you ever use your taste or judgement to go against what the algorithm said you should do?
We had guesses, but nothing particularly informed. My recollection is that we were right more often than not in our guesses, but it's been a while. We only ran experiments on changes we were willing to make. We never went against conclusive results due to judgement.
We also used data a bit less formally. For example, we had something similar to Google analytics' "funnel" feature of how people arrived at some of our pages, and we noticed a usage pattern that indicated we were missing a feature. When we added the feature directly it was one of our most used features.
Comments
I haven't done this in a while, but somewhere in my notes I have a formula to determine sample size based upon power and the effect we want to detect.
After we hit that size we would analyze the results. If they were ambiguous we just picked the one we subjectively felt was the better choice.
I'm guessing you would have have an idea of what the right answer was going to be before you ran the analysis. How often was your guess wrong? How often did the data tell you to do something that you were uncomfortable with? Did you ever use your taste or judgement to go against what the algorithm said you should do?
We had guesses, but nothing particularly informed. My recollection is that we were right more often than not in our guesses, but it's been a while. We only ran experiments on changes we were willing to make. We never went against conclusive results due to judgement.
We also used data a bit less formally. For example, we had something similar to Google analytics' "funnel" feature of how people arrived at some of our pages, and we noticed a usage pattern that indicated we were missing a feature. When we added the feature directly it was one of our most used features.