Ask HN: Success from A/B testing?

I am interested to know how many of you have initiated an A/B test of your home page and was it instrumental in helping you make decisions? We are considering a redesign and are thinking about an A/B test with the two different landing pages but I wanted to hear about your experiences first.