Polls are generally horrible. Self-selection bias, self-deception (i.e., what one perceives of one's self and what one actually does are almost always different and often wildly so), and self-disclosure bias (i.e., what one wants (others) to believe about one's self).
The entire analyst industry is basically built to do this. Of course, they tend to be ridiculously biased because they are getting paid so much for their opinions. Finding friends who work at places that already have the reports is one way to get them. :-)
In terms of actually useful data... Look for information about the users' and customers' actual behavior -- i.e., where the user/customer actually had to do something real like visit repeatedly, pay money, recommend you to other people, etc. As a number of folks mentioned at SUS, if you don't have information directly relevant to your play then look for various proxies and triangulate.
Polls don't have to be filled with questions about what you would do, leaving room for inaccurate self-prediction. The questions could be "have you in the past month done x" which tend to be a bit closer to reality.
Also, certain techniques (randomizing the order of the poll questions for each visitor) and statistical functions can be used to get more accurate information from those polled.
But you're right, if done improperly you can get wildly inaccurate data (just look at past or current presidential election polls)
True, you can setup the polls and question to reduce the problems but they are all still there since you still asking the user rather than measuring the actual behavior.
I.e., polls are a form of proxy for what you really want to learn but can't get directly. Your example of voting polls is a great example of this... Until the actual election, we can't know and so polling is about as good as we can get. (But I won't even bother to mention all of the games that are played with the polling to manipulate the outcomes. :-( )
Comments
Polls are generally horrible. Self-selection bias, self-deception (i.e., what one perceives of one's self and what one actually does are almost always different and often wildly so), and self-disclosure bias (i.e., what one wants (others) to believe about one's self).
The entire analyst industry is basically built to do this. Of course, they tend to be ridiculously biased because they are getting paid so much for their opinions. Finding friends who work at places that already have the reports is one way to get them. :-)
In terms of actually useful data... Look for information about the users' and customers' actual behavior -- i.e., where the user/customer actually had to do something real like visit repeatedly, pay money, recommend you to other people, etc. As a number of folks mentioned at SUS, if you don't have information directly relevant to your play then look for various proxies and triangulate.
Polls don't have to be filled with questions about what you would do, leaving room for inaccurate self-prediction. The questions could be "have you in the past month done x" which tend to be a bit closer to reality.
Also, certain techniques (randomizing the order of the poll questions for each visitor) and statistical functions can be used to get more accurate information from those polled.
But you're right, if done improperly you can get wildly inaccurate data (just look at past or current presidential election polls)
True, you can setup the polls and question to reduce the problems but they are all still there since you still asking the user rather than measuring the actual behavior.
I.e., polls are a form of proxy for what you really want to learn but can't get directly. Your example of voting polls is a great example of this... Until the actual election, we can't know and so polling is about as good as we can get. (But I won't even bother to mention all of the games that are played with the polling to manipulate the outcomes. :-( )