Exponential Distribution

Free Books Spectral Audio Signal Processing

Among probability distributions which are nonzero over a semi-infinite range of values $x\in[0,\infty]$ and having a finite mean $\mu$ , the exponential distribution has maximum entropy.

To the previous case, we add the new constraint

$\displaystyle \int_{-\infty}^\infty x\,p(x)\,dx = \mu < \infty$

(D.39)

resulting in the objective function

$\begin{eqnarray*} J(p) &\isdef & -\int_0^\infty p(x) \, \ln p(x)\,dx + \lambda_0\left(\int_0^\infty p(x)\,dx - 1\right).\\ & & + \lambda_1\left(\int_0^\infty x\,p(x)\,dx - \mu\right) \end{eqnarray*}$

Now the partials with respect to are

$\begin{eqnarray*} \frac{\partial}{\partial p(x)\,dx} J(p) &=& - \ln p(x) - 1 + \lambda_0 + \lambda_1 x\\ \frac{\partial^2}{\partial p(x)^2 dx} J(p) &=& - \frac{1}{p(x)} \end{eqnarray*}$

and is of the form $p(x) = e^{(\lambda_0-1)+\lambda_1x}$ . The unit-area and finite-mean constraints result in $\exp(\lambda_0-1) = 1/\mu$ and $\lambda_1=-1/\mu$ , yielding