I think that both those KPIs are quite a good idea, but they can alos lead to quite wrong measurements: What if this leads to people just clicking on every comment link to find out if there is a discussion or if there is a new post... etc.
It doesn't meassure the quality of the discussion and the submissions.
How many comments there are could be visible without any detrimental effect.
I simply suspect that the 'points' system has a shadow side and that as the site grows the shadow side starts to overpower the positive portion. By temporarily switching it off and asking the community what they felt about being 'point blind' for a short period and if they thought the quality improved or not you can make a 'metric' that is much easier to measure than some of the more technical tricks you could pull:
Customer Satisfaction.
A simple poll after the experiment would suffice. Or you could make it switcheable on a user basis if it is a toss-up or too close to call a very clear preference.
If on average more people feel better without the points visible than with them after a short trial period then it's something that you could consider doing permanently.
Another option would be to keep author and points hidden until after you've voted for a comment, but that may have other side effects.
I think the content should stand on its own, regardless of what the voting history and the author are it is what you think about it that counts.
Right now the choice to vote or not to vote is made plenty of time based on the current number of votes, which leads to plenty of feedback loops. I've seen 'flip-flops' (bi stable and tri-stable, 0,1 and -1,0,1), positive feedback loops, and negative feedback loops.
By breaking the loop we could end up with a more balanced view.
Imagine what the effect would be of a running tally during an election, it would completely affect the outcome, and not necessarily in a positive way.
Again, it's just a gut feeling but I think there is some truth to it, and it's a very easy to do experiment, worst case we will learn that it did not work.
I think the experiment would be a great idea and doing it the way you propose with empirical data to back up the machine generated statistics could provide very valuable insights.
Also one could start experimenting with the approach/idea e.g. let the number of votes appear after voting or not displaying a number at all
All this ultimately depends on what is the goal of HN. I believe the goal is to foster discussion. So, the number of comments would be a great metric.
Regarding clicks, I think even if people just click every comment link to find out what the discussion is about, the site/design has achieved its goal. It may reveal whether the points cloud new interesting articles.
Perhaps, this experiment could also include clouding of domain to see if techcrunch and other popular domains get extra juice thanks to its popularity.
One thing I've learned from A/B testing large sites is that you change only one metric at the time, but the domain clouding one sounds like an excellent candidate as well.
I agree that a given A/B test should change only one metric at a time. However I've had excellent results from running multiple A/B tests in parallel. As long as inclusion in each is independent and random, the results of each are informative, and if you're concerned about interaction effects you can analyze for signs of a potential interaction in a post-mortem, then do a more expensive multi-variate test if you have cause for concern.
Yep, that is right. Though you can do multivariate analysis, sticking with plain A/B test is best.
That said, you can (and should) measure performance on multiple benchmarks like clicks, comments, time spent. Gives you a correct picture of tradeoffs.
Surprisingly simple question that is very hard to answer well I think....
If you only test one variable at the time then you simplify your tests to the point where you can extract some metric to determine whether you've improved or not compared to the old situation.
Nothing stops you from then doing more ab testing with other combinations relative to your 'new best'. This may include going back to the original setting with some other variable changed, that way you avoid the local maximum problem.
So say we have a site in position 'A', we make version 'B' and we test them against each other. If we find out according to our chosen metric that 'B' performs better we now have several choices:
We can do another A/B test starting from 'B' changing some value to see if we can improve on 'B' directly, or alternatively we can go back to 'B' versus 'A' + some new modification that is not 'B'.
If you really believe that the A/B parameter space and the C/D parameter space interact with each other, then yes, you could get stuck in a local minimum. So in that case you should test all combinations simultaneously. However, it will take a lot longer to collect enough data in this case. So if you think it is likely that the parameters are independent, it would be better to change only one set at a time.
Comments
Interesting idea, but:
You have to worry about measurability. How would you measure the quality of the site? And how would you do it objectively?
Total number of comments and clicks on the articles would be the metric. Time spent on site would be a great measure too.
I don't think either of those are terribly good metrics.
Using two of my own submissions...
Compare Google's blog post on robots crawling news articles: http://news.ycombinator.com/item?id=708417 to an inflammatory post about Techcrunch: http://news.ycombinator.com/item?id=658308
One of those is 'quality', and the other has comments (and 'dead' status).
I think that both those KPIs are quite a good idea, but they can alos lead to quite wrong measurements: What if this leads to people just clicking on every comment link to find out if there is a discussion or if there is a new post... etc.
It doesn't meassure the quality of the discussion and the submissions.
How many comments there are could be visible without any detrimental effect.
I simply suspect that the 'points' system has a shadow side and that as the site grows the shadow side starts to overpower the positive portion. By temporarily switching it off and asking the community what they felt about being 'point blind' for a short period and if they thought the quality improved or not you can make a 'metric' that is much easier to measure than some of the more technical tricks you could pull:
A simple poll after the experiment would suffice. Or you could make it switcheable on a user basis if it is a toss-up or too close to call a very clear preference.If on average more people feel better without the points visible than with them after a short trial period then it's something that you could consider doing permanently.
Another option would be to keep author and points hidden until after you've voted for a comment, but that may have other side effects.
I think the content should stand on its own, regardless of what the voting history and the author are it is what you think about it that counts.
Right now the choice to vote or not to vote is made plenty of time based on the current number of votes, which leads to plenty of feedback loops. I've seen 'flip-flops' (bi stable and tri-stable, 0,1 and -1,0,1), positive feedback loops, and negative feedback loops.
By breaking the loop we could end up with a more balanced view.
Imagine what the effect would be of a running tally during an election, it would completely affect the outcome, and not necessarily in a positive way.
Again, it's just a gut feeling but I think there is some truth to it, and it's a very easy to do experiment, worst case we will learn that it did not work.
Thank you for further elaborating your idea.
I think the experiment would be a great idea and doing it the way you propose with empirical data to back up the machine generated statistics could provide very valuable insights.
Also one could start experimenting with the approach/idea e.g. let the number of votes appear after voting or not displaying a number at all
All this ultimately depends on what is the goal of HN. I believe the goal is to foster discussion. So, the number of comments would be a great metric.
Regarding clicks, I think even if people just click every comment link to find out what the discussion is about, the site/design has achieved its goal. It may reveal whether the points cloud new interesting articles.
Perhaps, this experiment could also include clouding of domain to see if techcrunch and other popular domains get extra juice thanks to its popularity.
One thing I've learned from A/B testing large sites is that you change only one metric at the time, but the domain clouding one sounds like an excellent candidate as well.
I agree that a given A/B test should change only one metric at a time. However I've had excellent results from running multiple A/B tests in parallel. As long as inclusion in each is independent and random, the results of each are informative, and if you're concerned about interaction effects you can analyze for signs of a potential interaction in a post-mortem, then do a more expensive multi-variate test if you have cause for concern.
Yep, that is right. Though you can do multivariate analysis, sticking with plain A/B test is best.
That said, you can (and should) measure performance on multiple benchmarks like clicks, comments, time spent. Gives you a correct picture of tradeoffs.
Doesn't changing only one metric at a time tend to lead to getting stuck in a local maximum?
Surprisingly simple question that is very hard to answer well I think....
If you only test one variable at the time then you simplify your tests to the point where you can extract some metric to determine whether you've improved or not compared to the old situation.
Nothing stops you from then doing more ab testing with other combinations relative to your 'new best'. This may include going back to the original setting with some other variable changed, that way you avoid the local maximum problem.
So say we have a site in position 'A', we make version 'B' and we test them against each other. If we find out according to our chosen metric that 'B' performs better we now have several choices:
We can do another A/B test starting from 'B' changing some value to see if we can improve on 'B' directly, or alternatively we can go back to 'B' versus 'A' + some new modification that is not 'B'.
I hope that's clear....
If you really believe that the A/B parameter space and the C/D parameter space interact with each other, then yes, you could get stuck in a local minimum. So in that case you should test all combinations simultaneously. However, it will take a lot longer to collect enough data in this case. So if you think it is likely that the parameters are independent, it would be better to change only one set at a time.