Indeed. This type of study, while really popular in the press, is often horribly confounded and hard to control for. I think the hacker news audience mostly knows to be skeptical, so I don't know why it was submitted.
I'll use this post as an excuse to mention the Lothian Birth Cohort. A really cool dataset of IQ test results at age 11 for (almost) all Scottish kids born in 1921 and 1936. While IQ tests are flawed in many ways, these data offer some means of control for follow up studies on these individuals. Many apparent effects disappear after controlling for age 11 IQ, for example: drinking coffee on intelligence — http://www.ncbi.nlm.nih.gov/pubmed/19995882 — nicely demonstrating the confounding problems affecting most studies.
Edit in response to sibling comment: yes, studies do try to control for these things. But it's hard, and often not really possible. This particular study might be ok, but we're not going to tell from a CNN article. We're also used to enough contradictory and over-hyped results in the popular press to know not to trust journalists to be a good filter.
In response to your edit: I think you're saying something very different than what your sibling responded to. The parent's claim was "They didn't correct for X"; your claim is "They tried to correct for X, but we have no way of knowing if they did it right."
The difference is that what you're saying is true of any study. There are always obvious things to control for, and less-obvious ones, and the authors always try and sometimes they do it right and sometimes they don't and there's never any way to tell for sure without redoing the study[0].
The fact that a layman reading a mainstream article about the study can come up with a possible confounding factor doesn't imply much if anything about its possible veracity. We should be equally suspicious of all such claims, not only those which happen to have flaws which are apparent to us.
[0] If the study contains a basic statistical error, redoing it might just be a matter of reinterpreting the original data; still counts.
Comments
Indeed. This type of study, while really popular in the press, is often horribly confounded and hard to control for. I think the hacker news audience mostly knows to be skeptical, so I don't know why it was submitted.
I'll use this post as an excuse to mention the Lothian Birth Cohort. A really cool dataset of IQ test results at age 11 for (almost) all Scottish kids born in 1921 and 1936. While IQ tests are flawed in many ways, these data offer some means of control for follow up studies on these individuals. Many apparent effects disappear after controlling for age 11 IQ, for example: drinking coffee on intelligence — http://www.ncbi.nlm.nih.gov/pubmed/19995882 — nicely demonstrating the confounding problems affecting most studies.
Edit in response to sibling comment: yes, studies do try to control for these things. But it's hard, and often not really possible. This particular study might be ok, but we're not going to tell from a CNN article. We're also used to enough contradictory and over-hyped results in the popular press to know not to trust journalists to be a good filter.
In response to your edit: I think you're saying something very different than what your sibling responded to. The parent's claim was "They didn't correct for X"; your claim is "They tried to correct for X, but we have no way of knowing if they did it right."
The difference is that what you're saying is true of any study. There are always obvious things to control for, and less-obvious ones, and the authors always try and sometimes they do it right and sometimes they don't and there's never any way to tell for sure without redoing the study[0].
The fact that a layman reading a mainstream article about the study can come up with a possible confounding factor doesn't imply much if anything about its possible veracity. We should be equally suspicious of all such claims, not only those which happen to have flaws which are apparent to us.
[0] If the study contains a basic statistical error, redoing it might just be a matter of reinterpreting the original data; still counts.