Benford's law solved with DSP

I have a longtime interest in the mystery of 1/f noise. A few years ago I came across Benford’s law, another puzzle that seemed to have many of the same characteristics.

Suppose you collect a large group of seemingly random numbers, such as might appear in a newspaper or financial report. Benford’s law relates to the leading digit of each number, such as "4" in 4.268, "3" in 0.0312, and "9" in -932.34. Since there are nine possible leading digits (1-9), you would expect that each of the leading digits would occur 1/9th of the time. The problem is, Nature doesn’t agree with your thinking-- the leading digit is a "1" over 30% of the time. This is hard to believe at first encounter, but it is absolutely true. And this is only the start-- there are other properties that seem simply bizarre.

The search to explain Benford’s law has a colorful history extending over more than 80 years. Explanations abound on the internet, from well respected mathematicians to kooks and clowns. One of the most interesting claims is that Benford’s law represents an underlying property of our existence, something akin to the paranormal. And this isn’t from the kooks, it's from the mathematicians! On balance, most mathematicians have viewed Benford’s law as a minor mystery, not a major problem to solve.

So, I started looking at Benford’s law in an attempt to understand 1/f noise. It shouldn’t be surprising that I used the tools I’m familiar with, convolution, Fourier analysis, and the like. The big surprise is that it worked unbelievably well. Signal processing provides an elegant solution to the mystery of Benford’s law.

Now you are probably wondering why I didn’t publish this result in some prominent journal. Well, I tried. No argument with the math, just not enough interest in the topic from professional mathematicians. They suggested it be submitted to a lesser known journal.

On reflection, I realized that there was something much more valuable in this work. The solution to Benford’s law is interesting in itself, but the way it is solved is far more important. Those in DSP are familiar with signals in the time domain. They are also familiar with signals in the spacial domain, such as images. The solution to Benford’s law shows how signal processing techniques can be applied to another important domain, the number line.

So I made the decision that I would forego traditional publication, and make the material available as a chapter of my on-line book. And here it is:

html: http://www.dspguide.com/ch34.htm

pdf (better): http://www.dspguide.com/CH34.PDF

Now the big question: Does the explanation of Benford’s law provide insight into 1/f noise? Not a damn bit. Back to where I started.

Steve Smith

2/22/2008