Maximum Entropy Distributions
Uniform Distribution
Among probability distributions
which are nonzero over a
finite range of values
, the maximum-entropy
distribution is the uniform distribution. To show this, we
must maximize the entropy,
![]() |
(D.33) |
with respect to


Using the method of Lagrange multipliers for optimization in the presence of constraints [86], we may form the objective function
![]() |
(D.34) |
and differentiate with respect to


![]() |
(D.35) |
Setting this to zero and solving for

![]() |
(D.36) |
(Setting the partial derivative with respect to

Choosing
to satisfy the constraint gives
, yielding
![]() |
(D.37) |
That this solution is a maximum rather than a minimum or inflection point can be verified by ensuring the sign of the second partial derivative is negative for all

![]() |
(D.38) |
Since the solution spontaneously satisfied

Exponential Distribution
Among probability distributions
which are nonzero over a
semi-infinite range of values
and having a finite
mean
, the exponential distribution has maximum entropy.
To the previous case, we add the new constraint
![]() |
(D.39) |
resulting in the objective function

Now the partials with respect to
are

and
is of the form
. The
unit-area and finite-mean constraints result in
and
, yielding
![]() |
(D.40) |
Gaussian Distribution
The Gaussian distribution has maximum entropy relative to all
probability distributions covering the entire real line
but having a finite mean
and finite
variance
.
Proceeding as before, we obtain the objective function

and partial derivatives

leading to
![]() |
(D.41) |
For more on entropy and maximum-entropy distributions, see [48].
Next Section:
Gaussian Mean
Previous Section:
Example: Random Bit String