A/B testing is not a tool, it's mental model seeking learnings and improvements
No, it's a validation phase most businesses use to sign off product decisions that are already made. They invest thousands in a new feature, run an A/B test, and then wait until they see an ambiguous 'win' in favour of the new feature. This is somewhat akin to flipping a coin until you get three heads in a row, and declaring yourself winner of the game.
You can denounce this as A/B testing gone wrong, but it's a reality of the web industry that most testing is badly motivated and statistically illiterate. The mentality I describe isn't limited to SMEs and tinpoint organisations, either. I've seen it espoused by highly respected product managers at prestigious organisations.
Where does this endemic of pseudoempiricism come from? Well, if I had to guess, I'd point to the marketing of companies like your own.
Take a look at https://vwo.com/ab-testing/ - "AB testing - The Complete Guide". There's not one mention of statistical significance either in word or spirit. There's no talk of setting a timeline in advance of the test. There's no hard numbers to set a context of what's an acceptable sample size - in fact the guide is specifically aimed at small business owners.
So if anyone is undermining trust in the "scientific process", it's not Optimizely, but in a way - you.
There's no talk of setting a timeline in advance of the test. There's no hard numbers to set a context of what's an acceptable sample size - in fact the guide is specifically aimed at small business owners.
You should take a trial of our product. Before setting up a test in VWO, we ensure you understand that the test will have to be run until we have statistically significant results. We've moved away from frequentist methodologies (where peeking at results was an issue) to bayesian. Check out what we do here: https://vwo.com/blog/smartstats-testing-for-truth/
The guide is aimed at people who are just starting out, but I agree with you, we should have pointed them to the importance of getting statistically correct results.
No, it's a validation phase most businesses use to sign off product decisions that are already made.
Interesting, could you elaborate further on this? I've only really worked with technically unsophisticated (i.e. non-tech companies), and they've always shown willing to do it properly and factor in results, and not set up leading experiments.
Is it a symptom of technical companies that staff are more bought-in to their solutions?
Comments
No, it's a validation phase most businesses use to sign off product decisions that are already made. They invest thousands in a new feature, run an A/B test, and then wait until they see an ambiguous 'win' in favour of the new feature. This is somewhat akin to flipping a coin until you get three heads in a row, and declaring yourself winner of the game.
You can denounce this as A/B testing gone wrong, but it's a reality of the web industry that most testing is badly motivated and statistically illiterate. The mentality I describe isn't limited to SMEs and tinpoint organisations, either. I've seen it espoused by highly respected product managers at prestigious organisations.
Where does this endemic of pseudoempiricism come from? Well, if I had to guess, I'd point to the marketing of companies like your own.
Take a look at https://vwo.com/ab-testing/ - "AB testing - The Complete Guide". There's not one mention of statistical significance either in word or spirit. There's no talk of setting a timeline in advance of the test. There's no hard numbers to set a context of what's an acceptable sample size - in fact the guide is specifically aimed at small business owners.
So if anyone is undermining trust in the "scientific process", it's not Optimizely, but in a way - you.
You should take a trial of our product. Before setting up a test in VWO, we ensure you understand that the test will have to be run until we have statistically significant results. We've moved away from frequentist methodologies (where peeking at results was an issue) to bayesian. Check out what we do here: https://vwo.com/blog/smartstats-testing-for-truth/
The guide is aimed at people who are just starting out, but I agree with you, we should have pointed them to the importance of getting statistically correct results.
Interesting, could you elaborate further on this? I've only really worked with technically unsophisticated (i.e. non-tech companies), and they've always shown willing to do it properly and factor in results, and not set up leading experiments.
Is it a symptom of technical companies that staff are more bought-in to their solutions?
That's not the fault of the tool if people want to throw money at it and ignore the results.
Please let people who want to throw money throw money, while the rest of us can use A/B testing to improve our sites. Thank you.