They mention in TFA that even surveying anomalies (people accidentally answering negatively) can’t be represented by a 99% figure.
That said, if I had a survey that asked a couple times in different ways if someone was satisfied with the iPhone and they gave conflicting answers, I’d probably simply throw their answer out for being nonsensical. Maybe that’s what’s going on here: they’re teasing out survey anomalies.
I'm the post author. I suspect aside from conducting extremely unrepresentative surveys, they are also aggressively filtering out respondents. I didn't include that since it is purely speculative on my part, but I'd guess that is part of what is happening.
I'd note that other seemingly innocuous things could have an impact too. For instance, if they surveyed only people who've owned their phone for 90 days or whatnot, you would eliminate the vast majority of people who didn't like it and returned it from the sample.
99% is an extraordinary claim, which is why we think Apple should be more transparent about their methodology.
Thanks for weighing in and your decisions in writing the article make sense to me.
I think it’s important (and probably simply ethical) for people and companies who make such extraordinary claims to explain how they obtained them, but having taken a stats class in college and doing a lot of growth-related work in my professional career I know how frustrating it is to get useful data, especially out of surveys.
Comments
They mention in TFA that even surveying anomalies (people accidentally answering negatively) can’t be represented by a 99% figure.
That said, if I had a survey that asked a couple times in different ways if someone was satisfied with the iPhone and they gave conflicting answers, I’d probably simply throw their answer out for being nonsensical. Maybe that’s what’s going on here: they’re teasing out survey anomalies.
I'm the post author. I suspect aside from conducting extremely unrepresentative surveys, they are also aggressively filtering out respondents. I didn't include that since it is purely speculative on my part, but I'd guess that is part of what is happening.
I'd note that other seemingly innocuous things could have an impact too. For instance, if they surveyed only people who've owned their phone for 90 days or whatnot, you would eliminate the vast majority of people who didn't like it and returned it from the sample.
99% is an extraordinary claim, which is why we think Apple should be more transparent about their methodology.
Thanks for weighing in and your decisions in writing the article make sense to me.
I think it’s important (and probably simply ethical) for people and companies who make such extraordinary claims to explain how they obtained them, but having taken a stats class in college and doing a lot of growth-related work in my professional career I know how frustrating it is to get useful data, especially out of surveys.