I don't much have a solution to the rabbit hole syndrome, but at the very least I can point out that your rating system could be solved by a confidence interval made using a Bernoulli random variable, which is a useful and (fortunately for you) a relatively simple formula to derive and comprehend.
I decided on a logarithmic weighted bayesian ranking. I read about the typical bayesian ranking (where you have some sort of minimal required number of reviews), then modified the it to work with power-curve distributed sets of data (as #reviews tends to be). Works great and has a few useful levers that I can adjust as I see fit.
Have you by any chance taken a look at the Wilson score? It's a modification of the typical Bayesian ranking that functions much better with small samples.
Comments
I don't much have a solution to the rabbit hole syndrome, but at the very least I can point out that your rating system could be solved by a confidence interval made using a Bernoulli random variable, which is a useful and (fortunately for you) a relatively simple formula to derive and comprehend.
I decided on a logarithmic weighted bayesian ranking. I read about the typical bayesian ranking (where you have some sort of minimal required number of reviews), then modified the it to work with power-curve distributed sets of data (as #reviews tends to be). Works great and has a few useful levers that I can adjust as I see fit.
Have you by any chance taken a look at the Wilson score? It's a modification of the typical Bayesian ranking that functions much better with small samples.