Sign in

username:

password:



Not a member?

Search Online Books



Search tips

Free Online Books



Chapters

See Also

Embedded SystemsFPGAElectronics

Spectral Audio Signal Processing
    Gaussian Function Properties
       Maximum Entropy Property of the Gaussian Distribution
          Maximum Entropy Distributions
             Uniform Distribution

Chapter Contents:

Search Spectral Audio Signal Processing

  

Book Index | Global Index

Would you like to be notified by email when Julius Orion Smith III publishes a new entry into his blog?

  

Uniform Distribution

Among probability distributions $ p(x)$ which are nonzero over a finite range of values $ x\in[a,b]$, the maximum-entropy distribution is the uniform distribution. To show this, we must maximize the entropy,

$\displaystyle H(p) \isdef -\int_a^b p(x)\, \lg p(x)\, dx $

with respect to $ p(x)$, subject to the constraints

\begin{eqnarray*} p(x) &\geq& 0\\ \int_a^b p(x)\,dx &=& 1. \end{eqnarray*}

Using the method of Lagrange multipliers for optimization in the presence of constraints [84], we may form the objective function

$\displaystyle J(p) \isdef -\int_a^b p(x) \, \ln p(x) \,dx + \lambda_0\left(\int_a^b p(x)\,dx - 1\right) $

and differentiate with respect to $ p(x)$ (and renormalize by dropping the $ dx$ factor multiplying all terms) to obtain

$\displaystyle \frac{\partial}{\partial p(x)\,dx} J(p) = - \ln p(x) - 1 + \lambda_0. $

Setting this to zero and solving for $ p(x)$ gives

$\displaystyle p(x) = e^{\lambda_0-1}. $

(Setting the partial derivative with respect to $ \lambda_0$ to zero merely restates the constraint.)

Choosing $ \lambda_0$ to satisfy the constraint gives $ \lambda_0 =1-\ln(b-a)$, yielding

$\displaystyle p(x) = \left\{\begin{array}{ll} \frac{1}{b-a}, & a\leq x \leq b \\ [5pt] 0, & \hbox{otherwise}. \\ \end{array}\right. $

That this solution is a maximum rather than a minimum or inflection point can be verified by ensuring the sign of the second partial derivative is negative for all $ x$:

$\displaystyle \frac{\partial^2}{\partial p(x)^2dx} J(p) = - \frac{1}{p(x)} $

Since the solution spontaneously satisfied $ p(x)>0$, it is a maximum.

Previous: Maximum Entropy Distributions
Next: Exponential Distribution

Order a Hardcopy of Spectral Audio Signal Processing

About the Author: Julius Orion Smith III
Julius Smith's background is in electrical engineering (BS Rice 1975, PhD Stanford 1983). He is presently Professor of Music and Associate Professor (by courtesy) of Electrical Engineering at Stanford's Center for Computer Research in Music and Acoustics (CCRMA), teaching courses and pursuing research related to signal processing applied to music and audio systems. See http://ccrma.stanford.edu/~jos/ for details.


Comments

No comments yet for this page


Add a Comment
You need to login before you can post a comment (best way to prevent spam). ( Not a member? )