That is your choice of course but it is classic Dunning Kruger. You are not required to form an opinion about concepts you are not au fait with and to do so reveals more about you than it does about the author.
I would read the Tom Nichols paper above; it changed the way I approach certain topics.
In all honesty though; I am struggling to see what further explanation you require?
In larger datasets the likelihood of spurious correlation increases. This is because in large data sets, large deviations are vastly more attributable to variance (or noise) than to information (or signal).
Figure 18 shows the swelling number of potential spurious relationships. The idea is as follows. If I have a set of 200 random variables, completely unrelated to each other, then it would be near impossible not to find in it a high correlation of sorts, say 30 percent, but that is entirely spurious. There are techniques to control the cherry-picking (one of which is known as the Bonferoni adjustment), but even then they don’t catch the culprits—much as regulation doesn’t stop insiders from gaming the system.
Thanks, the linked article gives a much better explanation.
It's not about my personal understanding of something, but about the apparent contradiction with other sources I trust and the unnecessary vagueness about a subject that requires precision in terminology. "Statistically robust but unlikely" has many meanings, but "spurious correlation" has only one. I am training to be an expert in mathematics, so I do feel somewhat entitled to discuss the finer (though perhaps overly pedantic) points. It's literally my job to rigorously reason about these things. So I notice that the word "large" means something very specific here (many independent variables), while you appear to still mean "lots of data," which can happen with few variables just as well. "Big data" encompasses both (volume and variety), which is why it's confusing. So my need for a further explanation is because "in large data sets large deviations happen more often" does not appear to address my question if I'm specifically asking why volume would cause more spurious correlations (in fact, it does not). My questions are admittedly vague as well, but this should clear up my confusion.
I'm not saying the author doesn't know what spurious correlation is, just that the author used the term incorrectly, or too vaguely to be called correct, in that instance. There are many reasons to do this intentionally, I'm sure, but as a consequence I will ask questions with trivial but technical answers. I ask a hundred such questions every day, most of which I can answer myself, but I'll continue to ask them even when (if ever) I'm considered an expert in any topic.
Comments
If you do not personally understand something you automatically assume that an author is incorrect?
http://thefederalist.com/2014/01/17/the-death-of-expertise/
That is your choice of course but it is classic Dunning Kruger. You are not required to form an opinion about concepts you are not au fait with and to do so reveals more about you than it does about the author.
I would read the Tom Nichols paper above; it changed the way I approach certain topics.
In all honesty though; I am struggling to see what further explanation you require?
In larger datasets the likelihood of spurious correlation increases. This is because in large data sets, large deviations are vastly more attributable to variance (or noise) than to information (or signal).
http://valbonneconsulting.files.wordpress.com/2013/10/traged...
Figure 18 shows the swelling number of potential spurious relationships. The idea is as follows. If I have a set of 200 random variables, completely unrelated to each other, then it would be near impossible not to find in it a high correlation of sorts, say 30 percent, but that is entirely spurious. There are techniques to control the cherry-picking (one of which is known as the Bonferoni adjustment), but even then they don’t catch the culprits—much as regulation doesn’t stop insiders from gaming the system.
http://www.wired.com/2013/02/big-data-means-big-errors-peopl...
Thanks, the linked article gives a much better explanation.
It's not about my personal understanding of something, but about the apparent contradiction with other sources I trust and the unnecessary vagueness about a subject that requires precision in terminology. "Statistically robust but unlikely" has many meanings, but "spurious correlation" has only one. I am training to be an expert in mathematics, so I do feel somewhat entitled to discuss the finer (though perhaps overly pedantic) points. It's literally my job to rigorously reason about these things. So I notice that the word "large" means something very specific here (many independent variables), while you appear to still mean "lots of data," which can happen with few variables just as well. "Big data" encompasses both (volume and variety), which is why it's confusing. So my need for a further explanation is because "in large data sets large deviations happen more often" does not appear to address my question if I'm specifically asking why volume would cause more spurious correlations (in fact, it does not). My questions are admittedly vague as well, but this should clear up my confusion.
I'm not saying the author doesn't know what spurious correlation is, just that the author used the term incorrectly, or too vaguely to be called correct, in that instance. There are many reasons to do this intentionally, I'm sure, but as a consequence I will ask questions with trivial but technical answers. I ask a hundred such questions every day, most of which I can answer myself, but I'll continue to ask them even when (if ever) I'm considered an expert in any topic.