Benford's law solved with DSP

Steve SmithFebruary 22, 20087 comments

I have a longtime interest in the mystery of 1/f noise. A few years ago I came across Benford’s law, another puzzle that seemed to have many of the same characteristics.

Suppose you collect a large group of seemingly random numbers, such as might appear in a newspaper or financial report. Benford’s law relates to the leading digit of each number, such as "4" in 4.268, "3" in 0.0312, and "9" in -932.34. Since there are nine possible leading digits (1-9), you would expect that each of the leading digits would occur 1/9th of the time. The problem is, Nature doesn’t agree with your thinking-- the leading digit is a "1" over 30% of the time. This is hard to believe at first encounter, but it is absolutely true. And this is only the start-- there are other properties that seem simply bizarre.

The search to explain Benford’s law has a colorful history extending over more than 80 years. Explanations abound on the internet, from well respected mathematicians to kooks and clowns. One of the most interesting claims is that Benford’s law represents an underlying property of our existence, something akin to the paranormal. And this isn’t from the kooks, it's from the mathematicians! On balance, most mathematicians have viewed Benford’s law as a minor mystery, not a major problem to solve.

So, I started looking at Benford’s law in an attempt to understand 1/f noise. It shouldn’t be surprising that I used the tools I’m familiar with, convolution, Fourier analysis, and the like. The big surprise is that it worked unbelievably well. Signal processing provides an elegant solution to the mystery of Benford’s law.

Now you are probably wondering why I didn’t publish this result in some prominent journal. Well, I tried. No argument with the math, just not enough interest in the topic from professional mathematicians. They suggested it be submitted to a lesser known journal.

On reflection, I realized that there was something much more valuable in this work. The solution to Benford’s law is interesting in itself, but the way it is solved is far more important. Those in DSP are familiar with signals in the time domain. They are also familiar with signals in the spacial domain, such as images. The solution to Benford’s law shows how signal processing techniques can be applied to another important domain, the number line.

So I made the decision that I would forego traditional publication, and make the material available as a chapter of my on-line book. And here it is:

html: http://www.dspguide.com/ch34.htm

pdf (better): http://www.dspguide.com/CH34.PDF

Now the big question: Does the explanation of Benford’s law provide insight into 1/f noise? Not a damn bit. Back to where I started.

Steve Smith


[ - ]
Comment by steph_tsfMarch 15, 2008
Fantastic constribution, as usual from Steve Smith. Would be interesting to see if a "genuine" audio .wav file recorded from nature sounds like wild life, human voice, or music, are following benford's law, when converted into decimal of floating point. Surely they do ! Or maybe not ? Benfor's law might be THE tool for detecting (alien) intelligence, but we are immediately losing the clue when digitizing the (alien) signal using 0/1 digital numbering scheme, isn't ? Or not ? Tere is maybe a thin chance for Benfor's law to become another tool for assessing audio equipment quality and fidelity. Thanks, Steve ! Steph
[ - ]
Comment by justpeterAugust 27, 2012
Thx so much for bringing me the privilege to disclose the Benford mystery in all its beauty. It is a shame that you had to bypass the math journals but you have succeeded exceptionally well. I am trying, with your paper as the final ammunition (although many willl never reach the details), to spread the importance of Benford towards improving everyday data processing and statistics in low-tech science. Great work!
[ - ]
Comment by steveuFebruary 24, 2008
That's s a really nice way of analysing the Benford rule/effect/not-really-a-law, as it homes right in on the rhythmic behaviour of working through successive decades. Although the result is basically just an artifact of using base 10 numbers, I understand it does seem to have practical application. I believe it is used to scan for fraud in financial markets. If someone selectively tampers with numbers, and doesn't appreciate the need to maintain the Benford behaviour, they might be spotted. However, I guess that will only home in on the less able. The smart guys know the numeric patterns they need to achieve to fool the watchdog.
[ - ]
Comment by JonTitusFebruary 8, 2009
Good to find your site. Benford's 30% rule triggered a look at my old Post slide rule. About 31% of the distance covers values between 1 and 2. The other 69% covers values from >2 to 10. Some companies use Benford's rule to monitor travel and entertainment expenses that help identify employees who make up random numbers for phony expenses.
[ - ]
Comment by raogApril 20, 2010
Yes, few people concern this matter. 1/f noise was focused. only "convolution" is reasonable I think. But why use DSP, only with paper and pencil is okay.
[ - ]
Comment by Sean O'ConnorDecember 7, 2012
Well it is sometimes forgotten that transistors are sophisticated quantum mechanical devices and that 1/f noise might be quantum background noise. I have a few different things to share with you:
1/ Regarding the design of analog filters that are equally as design flexible as digital filters I have EvoSpice 4.1 which is a numerical optimizer for LTSpice that is particularly good for filter design www.evospice.site88.net. There is a 4.2 version but I am promoting neither so grab the 4.1 version while you can.
2/ I have done a lot of work on the Walsh Hadamard transform, and some things you can do with it http://litetec.hubpages.com/hub/The-Walsh-Hadamard-Transform
3/ It is well worth looking up the Continuous Gray Code Optimization paper. I have further improvements on that.
[ - ]
Comment by zegaJanuary 21, 2014
In my head 1/f noise is little bit more complex then right part of the spectra from 1/f corner (broadband noise), since the part of bandwidths below the 1/f corner spans so low in freq. all the other effects get mixed into it, like offsets, drifts, sometimes induced by temperature changes, quantum stuff; all sort of slow stuff that we do not bluntly perceive but it's there...

To post reply to a comment, click on the 'reply' button attached to each comment. To post a new comment (not a reply to a comment) check out the 'Write a Comment' tab at the top of the comments.

Please login (on the right) if you already have an account on this platform.

Otherwise, please use this form to register (free) an join one of the largest online community for Electrical/Embedded/DSP/FPGA/ML engineers: