### Entropy of a Probability Distribution

The

*entropy*of a probability density function (PDF) is defined as [48]

(D.29) |

where denotes the logarithm base 2. The entropy of can be interpreted as the average number of bits needed to specify random variables drawn at random according to :

(D.30) |

The term can be viewed as the number of bits which should be assigned to the value . (The most common values of should be assigned the fewest bits, while rare values can be assigned many bits.)

**Next Section:**

Example: Random Bit String

**Previous Section:**

Binomial Distribution