Statistical significance? At what p-value? Controlled studies are great, but claims of "statistical significance" are not as strong as we make them to be. I also don't like this idea of an arbitrary threshold.
For the nice case of comparing two hypotheses, I'd rather talk about decibels of evidence: which hypothesis does it support, and how strongly does it support it. That way you don't even have to chose a null hypothesis. (You still have to work with some "priors", but in practice you have to assume things anyway).
Sure, use it to guide your instincts, but don't mistake it for rigorous science.
"Rigorous science" is the easy part. It's the point where you already have some insight or theory in mind, and you just have to test it. But first, you need to get that insight, or formulate your theory. At that point, your instincts is pretty much all you have.
Comments
Statistical significance? At what p-value? Controlled studies are great, but claims of "statistical significance" are not as strong as we make them to be. I also don't like this idea of an arbitrary threshold.
For the nice case of comparing two hypotheses, I'd rather talk about decibels of evidence: which hypothesis does it support, and how strongly does it support it. That way you don't even have to chose a null hypothesis. (You still have to work with some "priors", but in practice you have to assume things anyway).
"Rigorous science" is the easy part. It's the point where you already have some insight or theory in mind, and you just have to test it. But first, you need to get that insight, or formulate your theory. At that point, your instincts is pretty much all you have.