Yep, that is right. Though you can do multivariate analysis, sticking with plain A/B test is best.

That said, you can (and should) measure performance on multiple benchmarks like clicks, comments, time spent. Gives you a correct picture of tradeoffs.