Constructing experiments is hard, and interpreting data is even harder. PhDs, whose job it is to do these things, who typically have something on the order of a decade of education and even more years of practice doing these things, PhDs mess these things up. It happens all the time.
That's not even getting into the fact that in many times in data driven decision-making, an experiment is constructed to motivate a particular decision. If you have a preferred outcome, it's very easy to deliberately or unconsciously put your hand on the scale. Even without deliberate sabotage, there are many methods to apply to a particular dataset, and if you apply enough tools, sure enough one will support your assertion.
More often than not the result is that data driven decisionmaking is more like an elaborate ouija-board that says more about the people creating the experiments than what they purport to verify.
Agreed it's super hard. I think that's why you need to be careful about looking at data until you've got your hypothesis. Too much data and you'll overfit or introduce bias. I think of it more as a way to update your gut with an updated map of the state.
I'm guessing you would have have an idea of what the right answer was going to be before you ran the analysis. How often was your guess wrong? How often did the data tell you to do something that you were uncomfortable with? Did you ever use your taste or judgement to go against what the algorithm said you should do?
We had guesses, but nothing particularly informed. My recollection is that we were right more often than not in our guesses, but it's been a while. We only ran experiments on changes we were willing to make. We never went against conclusive results due to judgement.
We also used data a bit less formally. For example, we had something similar to Google analytics' "funnel" feature of how people arrived at some of our pages, and we noticed a usage pattern that indicated we were missing a feature. When we added the feature directly it was one of our most used features.
Comments
It's very hard to do anything in a data driven way. Where do you start? Where do you stop?
Is it signups that matter? Or DAUs? Or MAUs? Or (and it actually is this one) revenue?
But how do you decide the relationship between each of these? Does an easier on-board mean that you get less "sticky" users? How will you tell?
Yeah.
Constructing experiments is hard, and interpreting data is even harder. PhDs, whose job it is to do these things, who typically have something on the order of a decade of education and even more years of practice doing these things, PhDs mess these things up. It happens all the time.
That's not even getting into the fact that in many times in data driven decision-making, an experiment is constructed to motivate a particular decision. If you have a preferred outcome, it's very easy to deliberately or unconsciously put your hand on the scale. Even without deliberate sabotage, there are many methods to apply to a particular dataset, and if you apply enough tools, sure enough one will support your assertion.
More often than not the result is that data driven decisionmaking is more like an elaborate ouija-board that says more about the people creating the experiments than what they purport to verify.
Agreed it's super hard. I think that's why you need to be careful about looking at data until you've got your hypothesis. Too much data and you'll overfit or introduce bias. I think of it more as a way to update your gut with an updated map of the state.
I haven't done this in a while, but somewhere in my notes I have a formula to determine sample size based upon power and the effect we want to detect.
After we hit that size we would analyze the results. If they were ambiguous we just picked the one we subjectively felt was the better choice.
I'm guessing you would have have an idea of what the right answer was going to be before you ran the analysis. How often was your guess wrong? How often did the data tell you to do something that you were uncomfortable with? Did you ever use your taste or judgement to go against what the algorithm said you should do?
We had guesses, but nothing particularly informed. My recollection is that we were right more often than not in our guesses, but it's been a while. We only ran experiments on changes we were willing to make. We never went against conclusive results due to judgement.
We also used data a bit less formally. For example, we had something similar to Google analytics' "funnel" feature of how people arrived at some of our pages, and we noticed a usage pattern that indicated we were missing a feature. When we added the feature directly it was one of our most used features.