"Nobody seems sure why so much data has the Benford distribution. " : Have a look at http://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-... (cited in the previous discussion on HN) for an excellent exposition of the theory, and an explanation of the 'power' of the Benford Law.
That Terry Tao article is excellent, as usual (well, at least for math dorks - what's unusual about this one is that you don't need a very high level of dorkery to grok it 100%) and the "executive summary" reason that Benford's law applies so broadly is easy to find:
"More generally, it is not hard to show that if X obeys the continuous Benford’s law, and one multiplies X by some positive multiplier Y which is independent of the first digit of X (and, a fortiori, is independent of the fractional part of log_10(X)), one obtains another quantity X' = XY which also obeys the continuous Benford’s law."
In other words, multiplicative combinations of (independent) quantities will inherit Benford's law as long as any one of the quantities obeys it on its own, so it just takes one sub-factor that grows exponentially in order for an entire distribution to follow the law.
That line especially bothered me, it makes it seem like it's some divine or magical rule. If that were the case, I would hate for it to be used as if it were infallible with no actual understanding.
Comments
"Nobody seems sure why so much data has the Benford distribution. " : Have a look at http://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-... (cited in the previous discussion on HN) for an excellent exposition of the theory, and an explanation of the 'power' of the Benford Law.
That Terry Tao article is excellent, as usual (well, at least for math dorks - what's unusual about this one is that you don't need a very high level of dorkery to grok it 100%) and the "executive summary" reason that Benford's law applies so broadly is easy to find:
"More generally, it is not hard to show that if X obeys the continuous Benford’s law, and one multiplies X by some positive multiplier Y which is independent of the first digit of X (and, a fortiori, is independent of the fractional part of log_10(X)), one obtains another quantity X' = XY which also obeys the continuous Benford’s law."
In other words, multiplicative combinations of (independent) quantities will inherit Benford's law as long as any one of the quantities obeys it on its own, so it just takes one sub-factor that grows exponentially in order for an entire distribution to follow the law.
That line especially bothered me, it makes it seem like it's some divine or magical rule. If that were the case, I would hate for it to be used as if it were infallible with no actual understanding.