How do I calculate entropy?

How do I calculate entropy of a sampled signal of lenght N samples?

I assume I need the PDF (probability density function) - can I use the
Periodogram P(f) is this is like the histogram and use

H=-sum{=1 to N of } P(k)log(P(k))

does it have to be log base 2?

I assume if I take the entropy of white noise what  should be?

Also I notice that log(P(f) are the cepstrum coefficients.


Tom

Tom wrote:
> How do I calculate entropy of a sampled signal of lenght N samples?
> 
> I assume I need the PDF (probability density function) - can I use the
> Periodogram P(f) is this is like the histogram and use
> 
> H=-sum{=1 to N of } P(k)log(P(k))
> 
> does it have to be log base 2?
> 
> I assume if I take the entropy of white noise what  should be?
> 
> Also I notice that log(P(f) are the cepstrum coefficients.
> 
> 
> Tom

The entropy of a sequence X = { x_1  x_2  x_3 ... x_N } is defined as

   H(X) = sum_1_to_N ( p_i log( p_i ) )

     where p_i = | x_i | / || X ||

     and || X || = sum_1_to_N ( | x_i | ).

Good luck,
OUP

One Usenet Poster <me@my.computer.org> writes:

> Tom wrote:
> > How do I calculate entropy of a sampled signal of lenght N samples?
> > I assume I need the PDF (probability density function) - can I use
> > the
> 
> > Periodogram P(f) is this is like the histogram and use
> > H=-sum{=1 to N of } P(k)log(P(k))
> 
> > does it have to be log base 2?
> 
> > I assume if I take the entropy of white noise what  should be?
> 
> > Also I notice that log(P(f) are the cepstrum coefficients.
> 
> > Tom
> 
> 
> The entropy of a sequence X = { x_1  x_2  x_3 ... x_N } is defined as
> 
>    H(X) = sum_1_to_N ( p_i log( p_i ) )
> 
>      where p_i = | x_i | / || X ||
> 
>      and || X || = sum_1_to_N ( | x_i | ).

Hold on a minute there, cowboy. Where in the tarheeled' nation
did you get that? 

Nope. It oughta be

  H(X) = -1 * sum_1_to_N ( p_i log( p_i ) ),

where p_i is the probability of symbol x_i occurring and log is
base 2. 
-- 
Randy Yates
Sony Ericsson Mobile Communications
Research Triangle Park, NC, USA
randy.yates@sonyericsson.com, 919-472-1124

Randy Yates wrote:

> Hold on a minute there, cowboy. Where in the tarheeled' nation
> did you get that? 

"The Minimum Entropy Criterion of Range Alignment in ISAR Motion 
Compensation", Genyuan Wang and Zheng Bao, Radar 97, 16 Oct 1997, p. 
237, equation (4).

OUP

"One Usenet Poster" <me@my.computer.org> wrote in message
news:10jhcr4hs9sag86@corp.supernews.com...
> Randy Yates wrote:
>
> > Hold on a minute there, cowboy. Where in the tarheeled' nation
> > did you get that?
>
> "The Minimum Entropy Criterion of Range Alignment in ISAR Motion
> Compensation", Genyuan Wang and Zheng Bao, Radar 97, 16 Oct 1997, p.
> 237, equation (4).
>
> OUP
Thanks but that s eems to be for binary data - I meant a sampled waveform.DO
you suggest converting everything to binary first?

Thanks

Tom

The entropy describes the information contained in a source, and is measured
in terms of the symbol alphabet.

Ex: 3.15 bits/symbol or X (alphabet elements)/symbol.

So if you want to know the entropy of a sampled waveform you need to decide
first your coding alphabet.

For example if you consider 1024 levels and 1024 symbols you'll have.

H{X} = {-1/log{1024}}* Sum {log{p(Xi=i)}*p{Xi=i}

If you want a Real-Time Entropy you can use this aproximation:

for i=1.N
    for j=1.Lenght(Buffer)
        Buffer[j] = input;
    end
    P = Histogram(Buffer)/Length(Buffer); // You can
     // calculate the histogram and divide by the number of
    // samples to get an approximation of P(Xi=i).
    H(i) = {-1/log{Length Buffer}}* Sum {log{P(k)}*P(k)
end

regards,

David.

">
> for i=1.N
>     for j=1.Lenght(Buffer)
>         Buffer[j] = input;
>     end
>     P = Histogram(Buffer,Length{P})/Length(Buffer); // You can
>      // calculate the histogram and divide by the number of
>     // samples to get an approximation of P(Xi=i).
>     H(i) = {-1/log{Length Buffer}}* Sum {log{P(k)}*P(k)
> end

Sorry i meant H(i) = {-1/log{length(P)}}* Sum {log{P(k)}*P(k)}

Where P length is constant, you only changes the values contained in P

"David Martinez" <macarena.dupouy@neuf.fr> wrote in message
news:chbsi0$olb$1@apollon.grec.isp.9tel.net...
> The entropy describes the information contained in a source, and is
measured
> in terms of the symbol alphabet.
>
> Ex: 3.15 bits/symbol or X (alphabet elements)/symbol.
>
> So if you want to know the entropy of a sampled waveform you need to
decide
> first your coding alphabet.
>
> For example if you consider 1024 levels and 1024 symbols you'll have.
>
> H{X} = {-1/log{1024}}* Sum {log{p(Xi=i)}*p{Xi=i}
>
> If you want a Real-Time Entropy you can use this aproximation:
>
> for i=1.N
>     for j=1.Lenght(Buffer)
>         Buffer[j] = input;
>     end
>     P = Histogram(Buffer)/Length(Buffer); // You can
>      // calculate the histogram and divide by the number of
>     // samples to get an approximation of P(Xi=i).
>     H(i) = {-1/log{Length Buffer}}* Sum {log{P(k)}*P(k)
> end
>
> regards,
>
> David.
>
>
>
>
Thanks but I am thinking outside the circle a bit here - have a look at
this - I just found it.How does this fit in - am I asking the wrong
question?
http://www.menem.com/~ilya/digital_library/entropy/bercher_vignat_00.pdf

Tom