This article is really weakened by its failure to resolve the initial example.
What was it in the initial A/B test that resulted in your 10% improvement in conversion rate to be non-statistically significant?
Would applying a simple statistical test to your decision making process have resulted in a different conclusion?
There might be a deeper, useful insight in answering those questions, but as it stands now I really don't see anything valuable in this article. If anything it leads me to believe that the author is susceptible to p-hacking (see: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3204791)
Comments
This article is really weakened by its failure to resolve the initial example.
What was it in the initial A/B test that resulted in your 10% improvement in conversion rate to be non-statistically significant?
Would applying a simple statistical test to your decision making process have resulted in a different conclusion?
There might be a deeper, useful insight in answering those questions, but as it stands now I really don't see anything valuable in this article. If anything it leads me to believe that the author is susceptible to p-hacking (see: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3204791)