My favorite data-driven decision: one company I worked at ran a contest for the best-converting copy for the homepage. The secretary came in first place, the marketing guy came in last.
If you take one thing from the "agile startup" folks, take this: refactor your processes so that testing is not a "nice thing to do when we can" but is central to your business, such that you can scarcely imagine changing code without A/B testing it.
It is amazing how much more I A/B test now that it is one line to do. Previously, I'd go back and forth with myself about whether I should add a feature or not, worried whether the marginal utility would offset marginal complexity. I don't worry about these things anymore -- I just implement it and, toggle it with an A/B test (1 line of code), then track conversions at the next step in the funnel (1 line of code), and wait a week.
Concrete example: Next up on the development roadmap is a color selector. I've put it off for three years out of the fear that color selectors will befuddle my core user, but every few months someone asks for it. Why worry when you can implement it, wait for data, and rollback (almost) instantly if it doesn't pan out?
Writing a landing page? Write two calls to action. Takes essentially as much time as writing one since you mentally brainstormed six anyhow. No problem, chuck Mr. Second Best in there. It only takes a line. Customers will humble you with how many times Mr. Second Best trounces your best writing. (Customers will humble you with how many times Mr. Total Garbage trounces your best writing, too.)
Comments
You should add that NO landing page analysis is near complete without numbers from testing.
My experience with landing pages is that your deepest assumptions of what is better can be proven wrong by your visitors.
My favorite data-driven decision: one company I worked at ran a contest for the best-converting copy for the homepage. The secretary came in first place, the marketing guy came in last.
Is that study documented? Looks interesting..
The marketing guy ran the study, so he made sure that the results beyond 1st place were quickly buried.
You are right! I added a bit at the end. Reminder to always do testing when you can!
If you take one thing from the "agile startup" folks, take this: refactor your processes so that testing is not a "nice thing to do when we can" but is central to your business, such that you can scarcely imagine changing code without A/B testing it.
It is amazing how much more I A/B test now that it is one line to do. Previously, I'd go back and forth with myself about whether I should add a feature or not, worried whether the marginal utility would offset marginal complexity. I don't worry about these things anymore -- I just implement it and, toggle it with an A/B test (1 line of code), then track conversions at the next step in the funnel (1 line of code), and wait a week.
Concrete example: Next up on the development roadmap is a color selector. I've put it off for three years out of the fear that color selectors will befuddle my core user, but every few months someone asks for it. Why worry when you can implement it, wait for data, and rollback (almost) instantly if it doesn't pan out?
Writing a landing page? Write two calls to action. Takes essentially as much time as writing one since you mentally brainstormed six anyhow. No problem, chuck Mr. Second Best in there. It only takes a line. Customers will humble you with how many times Mr. Second Best trounces your best writing. (Customers will humble you with how many times Mr. Total Garbage trounces your best writing, too.)
Hey what do you use to do the A/B testing? Curious.
Thanks!
He uses this - http://www.bingocardcreator.com/abingo