Marissa ran an experiment where Google increased the number of search results to thirty. Traffic and revenue from Google searchers in the experimental group dropped by 20%.
Ouch. Why? Why, when users had asked for this, did they seem to hate it?
After a bit of looking, Marissa explained that they found an uncontrolled variable. The page with 10 results took .4 seconds to generate. The page with 30 results took .9 seconds.
Half a second delay caused a 20% drop in traffic. Half a second delay killed user satisfaction.
This feels like a situation where we found an answer, and extrapolated to say it must be the answer.
The article does at least have plenty of examples. I just can't help but think some of these are heavily confounded with "changes cause people to change." That is, I'd be curious to know if any of the tests did nothing other than increase latency.
The increase from 10 to 30 results, for example, had to have changed the look of the page. Would that alone have been enough to change behavior? My hypothesis is that it would be. Especially if there was a viable alternative that was still familiar to the user. Can't fathom by how much, though.
Comments
Yes, latency matters a lot in aggregate.
http://perspectives.mvdirona.com/2009/10/31/TheCostOfLatency...
From Marissa Mayer while at Google:
This feels like a situation where we found an answer, and extrapolated to say it must be the answer.
The article does at least have plenty of examples. I just can't help but think some of these are heavily confounded with "changes cause people to change." That is, I'd be curious to know if any of the tests did nothing other than increase latency.
The increase from 10 to 30 results, for example, had to have changed the look of the page. Would that alone have been enough to change behavior? My hypothesis is that it would be. Especially if there was a viable alternative that was still familiar to the user. Can't fathom by how much, though.