I believe it lets you compensate for the possibility that, say, all of your conversions might be coming from the bottom 1% of your users. Segmenting A into A1/A2 therefore insulates your interpretation of the results for A from being as heavily skewed.
Yes but in your A/B test you shouldn't be picking first half vs second half. Each visitor should be randomly assigned, so it should mitigate the problem you mentioned.
Comments
I'm not sure I understand - isn't that essentially an A/B test because 25 + 25 = 50?
I believe it lets you compensate for the possibility that, say, all of your conversions might be coming from the bottom 1% of your users. Segmenting A into A1/A2 therefore insulates your interpretation of the results for A from being as heavily skewed.
Yes but in your A/B test you shouldn't be picking first half vs second half. Each visitor should be randomly assigned, so it should mitigate the problem you mentioned.