I think part of the point abeppu is making is whether or not the two the statistics for each advertisment type are different. You see a difference of a 0.02% CTR but is that significant?
An analogy (from an interesting post I can't find) is tossing a coin for two samples sets of a 1000 times. In one sample I wear a read jumper and in another I wear a green jumper. I find that the green jumper gives me a 0.02% improvement in producing heads. Therefore I will always wear my green jumper in future when I toss a coin.
Obviously however this is just random error. A statistical analysis of CTR will tell you if the difference between the two advertisments you are observing is significant. The larger the samples size, as abeppu wrote, the greater the greater confidence in your results.
Comments
I think part of the point abeppu is making is whether or not the two the statistics for each advertisment type are different. You see a difference of a 0.02% CTR but is that significant?
An analogy (from an interesting post I can't find) is tossing a coin for two samples sets of a 1000 times. In one sample I wear a read jumper and in another I wear a green jumper. I find that the green jumper gives me a 0.02% improvement in producing heads. Therefore I will always wear my green jumper in future when I toss a coin.
Obviously however this is just random error. A statistical analysis of CTR will tell you if the difference between the two advertisments you are observing is significant. The larger the samples size, as abeppu wrote, the greater the greater confidence in your results.