### Maximum Entropy Distributions

#### Uniform Distribution

Among probability distributions
which are nonzero over a
*finite* range of values
, the maximum-entropy
distribution is the *uniform* distribution. To show this, we
must maximize the entropy,

(D.33) |

with respect to , subject to the constraints

Using the method of *Lagrange multipliers* for optimization in
the presence of constraints [86], we may form the
*objective function*

(D.34) |

and differentiate with respect to (and renormalize by dropping the factor multiplying all terms) to obtain

(D.35) |

Setting this to zero and solving for gives

(D.36) |

(Setting the partial derivative with respect to to zero merely restates the constraint.)

Choosing to satisfy the constraint gives , yielding

(D.37) |

That this solution is a maximum rather than a minimum or inflection point can be verified by ensuring the sign of the second partial derivative is negative for all :

(D.38) |

Since the solution spontaneously satisfied , it is a maximum.

#### Exponential Distribution

Among probability distributions
which are nonzero over a
*semi-infinite* range of values
and having a finite
mean
, the *exponential* distribution has maximum entropy.

To the previous case, we add the new constraint

(D.39) |

resulting in the objective function

Now the partials with respect to are

and is of the form . The unit-area and finite-mean constraints result in and , yielding

(D.40) |

#### Gaussian Distribution

The Gaussian distribution has maximum entropy relative to all
probability distributions covering the entire real line
but having a finite mean
and *finite
variance*
.

Proceeding as before, we obtain the objective function

and partial derivatives

leading to

(D.41) |

For more on entropy and maximum-entropy distributions, see [48].

**Next Section:**

Gaussian Mean

**Previous Section:**

Example: Random Bit String