Sign in

username:

password:



Not a member?

Search Online Books



Search tips

Free Online Books

Ads

Chapters

See Also

Embedded SystemsFPGAElectronics

Spectral Audio Signal Processing
    Gaussian Function Properties
       Maximum Entropy Property of the Gaussian Distribution
          Entropy of a Probability Distribution

Chapter Contents:

Search Spectral Audio Signal Processing

  

Book Index | Global Index

Would you like to be notified by email when Julius Orion Smith III publishes a new entry into his blog?

  

Entropy of a Probability Distribution

The entropy of a probability density function (PDF) $ p(x)$ is defined as [48]

$\displaystyle \zbox {h(p) \isdef \int_x p(x) \cdot \lg\left[\frac{1}{p(x)}\right] dx} $

where $ \lg$ denotes the logarithm base 2. The entropy of $ p(x)$ can be interpreted as the average number of bits needed to specify random variables $ x$ drawn at random according to $ p(x)$:

$\displaystyle h(p) = {\cal E}_p\left\{\lg \left[\frac{1}{p(x)}\right]\right\} $

The term $ \lg[1/p(x)]$ can be viewed as the number of bits which should be assigned to the value $ x$. (The most common values of $ x$ should be assigned the fewest bits, while rare values can be assigned many bits.)

Previous: Maximum Entropy Property of the Gaussian Distribution
Next: Example: Random Bit String

Order a Hardcopy of Spectral Audio Signal Processing

About the Author: Julius Orion Smith III
Julius Smith's background is in electrical engineering (BS Rice 1975, PhD Stanford 1983). He is presently Professor of Music and Associate Professor (by courtesy) of Electrical Engineering at Stanford's Center for Computer Research in Music and Acoustics (CCRMA), teaching courses and pursuing research related to signal processing applied to music and audio systems. See http://ccrma.stanford.edu/~jos/ for details.


Comments

No comments yet for this page


Add a Comment
You need to login before you can post a comment (best way to prevent spam). ( Not a member? )