I'm not sure how well this will work. Users care about accessing content, not answering surveys.
A user can achieve their goal faster by clicking the top result instead of reading the question. Browser extensions like AdBlock can make the process even 'better' by auto-completing the surveys.
Correcting for these biases is problematic, especially if the user's true answer to the question correlates with things like AdBlock usage or patience.
It's been too long since I've taken statistics to be sure, but I wonder if you could account for this by randomly placing the options in the top position then comparing the results of when A is on top versus when B is on top?
Yes, that's a very good idea (and actually the one I had in mind when I talked about correcting the bias). The real problem is when avoidance of surveys correlates with one of the survey answers, like "Should we start using micro surveys?".
Comments
I'm not sure how well this will work. Users care about accessing content, not answering surveys.
A user can achieve their goal faster by clicking the top result instead of reading the question. Browser extensions like AdBlock can make the process even 'better' by auto-completing the surveys.
Correcting for these biases is problematic, especially if the user's true answer to the question correlates with things like AdBlock usage or patience.
It's been too long since I've taken statistics to be sure, but I wonder if you could account for this by randomly placing the options in the top position then comparing the results of when A is on top versus when B is on top?
Yes, that's a very good idea (and actually the one I had in mind when I talked about correcting the bias). The real problem is when avoidance of surveys correlates with one of the survey answers, like "Should we start using micro surveys?".