I poked around a bit on 538 on the day it relaunched, but hadn't read anything from there since then. Is this representative of the quality of articles? It seems just embarrassing as an example of data-driven journalism. They are evaluating the quality of a "big data" prediction algorithm based on 32 samples. The experimental setup is very odd, with such a short period of price tracking, a bizarre baseline strategy, and ending the trials early (over half right at the beginning!).
Or consider this gem: "For the five routes in which there was a financial benefit to waiting, Kayak successfully reduced my fare in each instance. [...] This is a sure sign of intelligence". Yes, of course if you cherry-pick 1/6 of the trials (a whole 5 of them!), the system will look effective. What kind of value is there in "analysis" of this level?
I've been checking out fivethirtyeight every few days. You are correct. The quality has gone down since relaunching. I believe this is due to the wider focus and the need to pump out articles. They still do some decent sports articles (and I assume the election prediction stuff will still be good), but overall, I'm not impressed.
I was underwhelmed as well. In addition to the items you point out, at the end the article says:
"...it still might be worth your time and energy to use airfare prediction software. Why? Because following the algorithm isn’t going to cost you more money..."
which is simply wrong, since the article explicitly gives examples of following the algorithm costing the author more money.
I think the author was basing that statement on his previous assertion that:
If weighted equally, the prices paid, following Kayak’s algorithm, were 2 percent higher than the initial March 29 prices. Wearing my statistical hat, I’d call that a tie between the two strategies.
The author also claims that:
in user surveys that in addition to appealing to quantitative types, another group of regular users said Farecast gave them “peace of mind.”
So if the price difference is 2% (basically a wash either way), then waiting would be a superior strategy because it "might actually relieve some of the second-guessing that occurs when you’re left to your own devices," and that peace of mind may be worth more than 2% of the ticket cost.
...or at least, I think that's the author's point.
I think that there's a lot of value in amateur data science from an "eyeballs" perspective. Young people want more "intelligent" coverage, but not so dense that they can't understand it. The sort of nitpicking you're engaging in suggests that you're not the target audience.
This is why I'm looking forward to more and more journals becoming open access. I think that for people who are dissatisfied with 538, understanding academic publications would not be too big of a leap.
You should check out this analysis of their analysis of their story on Bechdel-test-passing movies. It highlights some of the inadequacies in their methodologies and analysis and gives a call for more openness and higher scientific standards.
Note that early termination rules are pretty common in bayesian statistics, as is taking only enough samples for a reasonable likelihood ratio.
I'd rather see samples covering different time periods as well as different routes, but I'm perfectly happy updating my belief in the efficacy of Kayak's price prediction based on these results.
I'm giving it the 6 months that it took Grantland to figure themselves out before making a call on the overall quality - The Value of a Steal series has, so far, been the only really high quality piece I've read, everything else sounds like a proposal to study something interesting (or the abstract of the paper studying something interesting).
I think one of the major areas they might struggle in is decent prose. I expected the new 538 to have lengthy and interesting prose in addition to data, but most everything has been a slog to read so far. Maybe I've just been spoiled by Grantland, but I really hope it improves.
Yea, that's true. Grantland has truly excellent writers, whereas 538 may have been looking for something different than pure writing ability. Still, guys like Kirk Goldsberry could easily write for 538 in terms of technical competence.
Comments
I poked around a bit on 538 on the day it relaunched, but hadn't read anything from there since then. Is this representative of the quality of articles? It seems just embarrassing as an example of data-driven journalism. They are evaluating the quality of a "big data" prediction algorithm based on 32 samples. The experimental setup is very odd, with such a short period of price tracking, a bizarre baseline strategy, and ending the trials early (over half right at the beginning!).
Or consider this gem: "For the five routes in which there was a financial benefit to waiting, Kayak successfully reduced my fare in each instance. [...] This is a sure sign of intelligence". Yes, of course if you cherry-pick 1/6 of the trials (a whole 5 of them!), the system will look effective. What kind of value is there in "analysis" of this level?
I've been checking out fivethirtyeight every few days. You are correct. The quality has gone down since relaunching. I believe this is due to the wider focus and the need to pump out articles. They still do some decent sports articles (and I assume the election prediction stuff will still be good), but overall, I'm not impressed.
I was underwhelmed as well. In addition to the items you point out, at the end the article says:
"...it still might be worth your time and energy to use airfare prediction software. Why? Because following the algorithm isn’t going to cost you more money..."
which is simply wrong, since the article explicitly gives examples of following the algorithm costing the author more money.
I think the author was basing that statement on his previous assertion that:
The author also claims that:
So if the price difference is 2% (basically a wash either way), then waiting would be a superior strategy because it "might actually relieve some of the second-guessing that occurs when you’re left to your own devices," and that peace of mind may be worth more than 2% of the ticket cost.
...or at least, I think that's the author's point.
http://krugman.blogs.nytimes.com/2014/03/23/tarnished-silver...
I think that there's a lot of value in amateur data science from an "eyeballs" perspective. Young people want more "intelligent" coverage, but not so dense that they can't understand it. The sort of nitpicking you're engaging in suggests that you're not the target audience.
This is why I'm looking forward to more and more journals becoming open access. I think that for people who are dissatisfied with 538, understanding academic publications would not be too big of a leap.
You should check out this analysis of their analysis of their story on Bechdel-test-passing movies. It highlights some of the inadequacies in their methodologies and analysis and gives a call for more openness and higher scientific standards.
http://nbviewer.ipython.org/github/brianckeegan/Bechdel/blob...
Note that early termination rules are pretty common in bayesian statistics, as is taking only enough samples for a reasonable likelihood ratio.
I'd rather see samples covering different time periods as well as different routes, but I'm perfectly happy updating my belief in the efficacy of Kayak's price prediction based on these results.
I'm giving it the 6 months that it took Grantland to figure themselves out before making a call on the overall quality - The Value of a Steal series has, so far, been the only really high quality piece I've read, everything else sounds like a proposal to study something interesting (or the abstract of the paper studying something interesting).
I think one of the major areas they might struggle in is decent prose. I expected the new 538 to have lengthy and interesting prose in addition to data, but most everything has been a slog to read so far. Maybe I've just been spoiled by Grantland, but I really hope it improves.
Yea, that's true. Grantland has truly excellent writers, whereas 538 may have been looking for something different than pure writing ability. Still, guys like Kirk Goldsberry could easily write for 538 in terms of technical competence.