Reading "Thinking, fast and slow" right now, it resonates strangely with this article.
For those not in the knowing, this book proves how most of our reasoning is false most of the time, shows that people vote based on facial features, that success is luck, that the world is out of our control, that experts are worse forecasters than monkeys randomly picking options, and the better the experts the worse the forecast. Depressing.
Edit: one thing reassuring is that it explain rationally how random people moderately clever and sometime plain despicable can become the head of very successful companies.
I second the recommendation for Thinking, Fast and Slow.
> that experts are worse forecasters than monkeys randomly picking options
For a more nuanced study on this, see Expert Political Judgement: How Good Is It? How Can We Know? by Phillip Tetlock.
If you aggregate expert opinions then yes, they barely beat monkeys. Happily they outperform aggregated non-expert opinions.
However you can classify how experts reach conclusions and find that some categories of thinkers (foxes) will consistently outperform other categories (hedgehogs).
Mind you -- humans and monkeys both are absolutely trounced by statistical models.
It comes from an essay by Isaiah Berlin, The Hedgehog and the Fox. Hedgehogs know "one big thing" and Foxes know "lots of little things".
In the expert prediction study, experts who had a universal framework which they extended rigidly to every problem performed poorly on both calibration (correlation of confidence and actual post-factual probabilities) and discrimination (correlation of more/less/same predictions to actual outcomes).
Foxes tended to throw together a bunch of thoughts, including from incompatible theories, express a lot of "one of the one hand, and on the other" sort of thinking and qualified most of their predictions. They did much better on calibration and somewhat better on discrimination.
Basically, Hedgehogs tend to make more extreme predictions ("Russia will fall into civil war by 1997") more confidently ("it's almost certain") than Foxes. But statistically, about 40% of the time variables in the complex systems being forecast actually stayed within a band and neither decisively moved up or down.
I know that you're probably thinking of holes and objections to the study already -- read the book. Tetlock and his collaborators were astonishingly thorough in trying to deal with all the relevant arguments and counterarguments with actual data.
> I know that you're probably thinking of holes and objections to the study already
Quite the contrary. Your description of the differences is clear and the outcomes from the study fit well with a similar meme in machine learning around ensemble methods.
Comments
Reading "Thinking, fast and slow" right now, it resonates strangely with this article.
For those not in the knowing, this book proves how most of our reasoning is false most of the time, shows that people vote based on facial features, that success is luck, that the world is out of our control, that experts are worse forecasters than monkeys randomly picking options, and the better the experts the worse the forecast. Depressing.
Edit: one thing reassuring is that it explain rationally how random people moderately clever and sometime plain despicable can become the head of very successful companies.
I second the recommendation for Thinking, Fast and Slow.
> that experts are worse forecasters than monkeys randomly picking options
For a more nuanced study on this, see Expert Political Judgement: How Good Is It? How Can We Know? by Phillip Tetlock.
If you aggregate expert opinions then yes, they barely beat monkeys. Happily they outperform aggregated non-expert opinions.
However you can classify how experts reach conclusions and find that some categories of thinkers (foxes) will consistently outperform other categories (hedgehogs).
Mind you -- humans and monkeys both are absolutely trounced by statistical models.
Can you briefly summarize what distinguishes foxes vs. hedgehogs?
It comes from an essay by Isaiah Berlin, The Hedgehog and the Fox. Hedgehogs know "one big thing" and Foxes know "lots of little things".
In the expert prediction study, experts who had a universal framework which they extended rigidly to every problem performed poorly on both calibration (correlation of confidence and actual post-factual probabilities) and discrimination (correlation of more/less/same predictions to actual outcomes).
Foxes tended to throw together a bunch of thoughts, including from incompatible theories, express a lot of "one of the one hand, and on the other" sort of thinking and qualified most of their predictions. They did much better on calibration and somewhat better on discrimination.
Basically, Hedgehogs tend to make more extreme predictions ("Russia will fall into civil war by 1997") more confidently ("it's almost certain") than Foxes. But statistically, about 40% of the time variables in the complex systems being forecast actually stayed within a band and neither decisively moved up or down.
I know that you're probably thinking of holes and objections to the study already -- read the book. Tetlock and his collaborators were astonishingly thorough in trying to deal with all the relevant arguments and counterarguments with actual data.
> I know that you're probably thinking of holes and objections to the study already
Quite the contrary. Your description of the differences is clear and the outcomes from the study fit well with a similar meme in machine learning around ensemble methods.