Skip to content

Comment on What's wrong with social science and how to fix itparent

Comments

Here’s the paper, published only recently: https://royalsocietypublishing.org/doi/10.1098/rsos.200566

I bet you could do some ML to predict replicability of a paper (per author's suggestion that it's laughably easy to predict)

I am betting any such ML system could be gamed and addressing the issue would ultimately still need humans in the loop. For example, what if I am selective with my data, beyond the visibility of ML evaluating the final published paper? I don’t think this is “laughably easy” to predict. It may be easy to spot telltale signs today that predict replicability, but as soon as those markers are understood, I imagine authors will simply squeeze papers through the cracks in a different way.

Another issue is this bit from the author on Twitter:

Just because it replicates doesn't mean it's good. A replication of a badly designed study is still badly designed. There are tons of papers doing correlational analyses yet drawing causal conclusions, and many of them will successfully replicate. Doesn't mean they're justified.
AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.