Let’s be honest: that last sentence was far too generous. Many of us engaged in practices that, in hindsight, were borderline dishonest. We abused experimenter degrees of freedom, engaged in questionable research practices, p-hacked, massaged our data—you pick the euphemism. In contrast, this new replication study followed the most up-to-date best practices in psychological science, eliminating room for flexibility in analysis or results interpretation.
Exploiting researcher degrees of freedom remains unfortunately extremely common. There needs to be some sort of statistical vanguard in the ivory towers enforcing real preregistration and good analysis practices. Strict epistemic discipline is necessary to do real science.
The real problem is we have a lot of data. So it is easy to shove it all into a computer and then see what comes out. This is not a bad thing, but it isn't alone a result to publish it is instead justification to do a real study on whatever thing statistics turns up as a significant result to make sure it really is when you properly isolate for that data.
There’s also the issue that in psych, everything is measured through indirect proxies and most of the predictions will be directional. That gives experimenters a LOT of wiggle room.
In physical science you can often directly measure a phenomenon and have a theory that makes very specific predictions
early in my psych career, i was actively encouraged to do things that would be considered p-hacking or abusing degrees of freedom. it was seen as a way to not let any data go to waste and potentially 'show off' your cleverness/mathematical chops in the methods section (anybody can just run an ANOVA like they teach in the text book, but look what I can do).
Comments
Exploiting researcher degrees of freedom remains unfortunately extremely common. There needs to be some sort of statistical vanguard in the ivory towers enforcing real preregistration and good analysis practices. Strict epistemic discipline is necessary to do real science.
The real problem is we have a lot of data. So it is easy to shove it all into a computer and then see what comes out. This is not a bad thing, but it isn't alone a result to publish it is instead justification to do a real study on whatever thing statistics turns up as a significant result to make sure it really is when you properly isolate for that data.
There’s also the issue that in psych, everything is measured through indirect proxies and most of the predictions will be directional. That gives experimenters a LOT of wiggle room.
In physical science you can often directly measure a phenomenon and have a theory that makes very specific predictions
early in my psych career, i was actively encouraged to do things that would be considered p-hacking or abusing degrees of freedom. it was seen as a way to not let any data go to waste and potentially 'show off' your cleverness/mathematical chops in the methods section (anybody can just run an ANOVA like they teach in the text book, but look what I can do).