Entropy of a Probability Distribution
The entropy of a probability density function (PDF) 
is
defined as [48]
![$\displaystyle \zbox {h(p) \isdef \int_x p(x) \cdot \lg\left[\frac{1}{p(x)}\right] dx}$](http://www.dsprelated.com/josimages_new/sasp2/img2794.png) |
(D.29) |
where
denotes the logarithm base 2. The entropy of
can
be interpreted as the average number of bits needed to specify random
variables 
drawn at random according to
:
![$\displaystyle h(p) = {\cal E}_p\left\{\lg \left[\frac{1}{p(x)}\right]\right\}$](http://www.dsprelated.com/josimages_new/sasp2/img2796.png) |
(D.30) |
The term
![$ \lg[1/p(x)]$](http://www.dsprelated.com/josimages_new/sasp2/img2797.png){kind=small}
can be viewed as the number of bits which should
be assigned to the value
. (The most common values of
should
be assigned the fewest bits, while rare values can be assigned many
bits.)
Next Section:
Example: Random Bit String
Previous Section:
Binomial Distribution
