The main thing I got out of this survey is that is possible to design an objective-sounding survey that will favor any brand that you choose.
The "Smart Rating" he uses factors in various critical review sites at various arbitrary weightings (eg., TechCrunch at 13.9%) and various specs at various arbitrary weightings (eg., cable length at 0.7%).
If I wanted to cheat and create a survey that favored brand X, I could choose and tweak parameters until X came out on top. For example, I know that brand X always has nice long cables, so I'll give cable length a higher smart score. I know that Wired has reviewed some of the crappy models of brand X, and hated them, so I'll drop Wired from my formula. PC Mag however has reviewed the best model that brand X makes, and loved it, so I'll add PC Mag to the mix. Etc.
I'm not saying the author rigged anything. I'm sure it's his earnest effort to create a meaningful measure for headphones. It's just that the formula looks objective and authoritative, but another formula would've given you any result you desire.
The best thing about the article was it got Hacker News readers talking about headphones! Now I know I'm getting real scoop from users who likely are using the phones in a similar way to myself.
Comments
The main thing I got out of this survey is that is possible to design an objective-sounding survey that will favor any brand that you choose.
The "Smart Rating" he uses factors in various critical review sites at various arbitrary weightings (eg., TechCrunch at 13.9%) and various specs at various arbitrary weightings (eg., cable length at 0.7%).
If I wanted to cheat and create a survey that favored brand X, I could choose and tweak parameters until X came out on top. For example, I know that brand X always has nice long cables, so I'll give cable length a higher smart score. I know that Wired has reviewed some of the crappy models of brand X, and hated them, so I'll drop Wired from my formula. PC Mag however has reviewed the best model that brand X makes, and loved it, so I'll add PC Mag to the mix. Etc.
I'm not saying the author rigged anything. I'm sure it's his earnest effort to create a meaningful measure for headphones. It's just that the formula looks objective and authoritative, but another formula would've given you any result you desire.
Agreed. The methodology was pretty weak.
The best thing about the article was it got Hacker News readers talking about headphones! Now I know I'm getting real scoop from users who likely are using the phones in a similar way to myself.