information theory?

On Nov 2, 2:51&#4294967295;pm, RichD <r_delaney2...@yahoo.com> wrote:
> On Nov 2, Chris <chris.sant...@gmail.com> wrote:
>
> > Be careful going down this path. For me it was
> > much more interesting in theory than in practice.
>
> > Me in 2008: "Like, what *is* the nature of information,
> > man?" (waves hand to clear smoke from the room)
>
> > Then I signed up for Info Theory, and I spent
> > a whole semester taking integrals of logs of
> > probability distributions and proving that one
> > was bounded by the other.
>
> It's too bad you got so little out of it.
> It's not 'just a theory', it's a field
> where theory and practice coincide.
> The theorists concoct the recipes, the
> engineers follow the instructions, and
> it works, bang on. &#4294967295;Fukengrooven.
> And it's the root of many a Silly Con
> Valley fortune.
>
> I strongly suggest, if you have some
> free neurons and time and nothing's good on
> tv, read Shannon's original paper. &#4294967295;It's
> utterly fucking brilliant. &#4294967295;There's no doubt,
> in Valhalla, Shannon shares a table with Einstein.
>
> It's sort of funny, he called it communication
> theory, it was his disciples who popularized
> 'information theory'.
>
> --
> Rich

Oh, I'm not saying I got little from it. On the contrary. It comes up
all the time. I'm just saying that the nitty gritty details can be
painfully boring to work though when disconnected from cool
applications.

On Nov 2, 2:51&#4294967295;pm, RichD <r_delaney2...@yahoo.com> wrote:
> On Nov 2, Chris <chris.sant...@gmail.com> wrote:
>
> > Be careful going down this path. For me it was
> > much more interesting in theory than in practice.
>
> > Me in 2008: "Like, what *is* the nature of information,
> > man?" (waves hand to clear smoke from the room)
>
> > Then I signed up for Info Theory, and I spent
> > a whole semester taking integrals of logs of
> > probability distributions and proving that one
> > was bounded by the other.
>
> It's too bad you got so little out of it.
> It's not 'just a theory', it's a field
> where theory and practice coincide.
> The theorists concoct the recipes, the
> engineers follow the instructions, and
> it works, bang on. &#4294967295;Fukengrooven.
> And it's the root of many a Silly Con
> Valley fortune.
>
> I strongly suggest, if you have some
> free neurons and time and nothing's good on
> tv, read Shannon's original paper. &#4294967295;It's
> utterly fucking brilliant. &#4294967295;There's no doubt,
> in Valhalla, Shannon shares a table with Einstein.
>
> It's sort of funny, he called it communication
> theory, it was his disciples who popularized
> 'information theory'.
>
> --
> Rich

One of the coolest ideas I remember learning about is directed
divergence:

http://en.wikipedia.org/wiki/Kullback&#4294967295;Leibler_divergence

and being able to estimate how much it will cost you in bits if your
estimate of the underlying probability distribution of the information
you're trying to code wrong. I'd cite the fact that - even though I'm
not a super expert on all the topics - the fact that I can pick up
these topics again rather quickly as evidence that dipping my toes in
the field was worth it. School's not about becoming an expert - it's
about working the muscles so you can become an expert if you need to.

On Nov 3, 12:58&#4294967295;pm, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> In sci.crypt Les Cargill <lcargil...@comcast.com> wrote:
>
>
>
> > Penroses' "Emperor's New Mind" misses quite a bit ( he skyhooks some
> > stuff ) but that alone should show they're thinking about it.
>
> _The Road to Reality_ goes into a lot of detail.
>
> Andrew.

The truth will set you free, but first it will make you look stupid.

- gaby de wilde

On Wed, 02 Nov 2011 22:40:13 -0400, Christopher J. Henrich wrote:

> In article <DP-dnaStmoSaeSzTnZ2dnUVZ5oadnZ2d@giganews.com>, Lofty Goat
> <rlwatkins@gmail.com> wrote:
> 
> 
>> But the earliest I can remember someone doing otherwise, in earnest, is
>> 1977, with the publication of Kantor's "Information Mechanics".  There
>> are probably earlier examples.

> I am delighted to learn that somebody besides me knows about that book.
> What do you think of it?

While hesitant to offer an opinion after thirty years, what it said, 
indirectly, about the limits and potential of computational machinery 
seemed most interesting, something which seems now to be coming full 
circle, as well as what seemed a pretty novel approach, at the time, to 
reasoning about the World Of Quanta.

And I'll confess that I've been away from physics so long that if I were 
to re-read it I'd probably not understand but a fraction.  You'll find 
others hereabouts I'm sure whose judgments will be more useful.

-- 
RLW

On Nov 2, VWWall <vw...@large.invalid> wrote:
> > I strongly suggest, if you have some
> > free neurons and time and nothing's good on
> > tv, read Shannon's original paper. &#4294967295;It's
> > utterly fucking brilliant. &#4294967295;There's no doubt,
> > in Valhalla, Shannon shares a table with Einstein.
>
> I was at Bell Labs in 1949, and Betty Moore, Claude
> Shannon's wife, worked in the same department.

Holy Methuselah, Batman!  Did you get to shake
Abe Lincoln's hand, too?

> > It's sort of funny, he called it communication
> > theory, it was his disciples who popularized
> > 'information theory'.
>
> Shannon took a very practical outlook on "information".
> I recall how he would stop people in the halls and ask
> them to guess the next letter in a sentence he'd show
 >them. &#4294967295;I remember one as: "A motorcycle has no
> reverse; it can not back up." &#4294967295;He used this method
> to determine the redundancy in the English language.

He estimated music contains 40 bits/second entropy.
How close is MP3 to that?

--
Rich

On 11/3/2011 11:54 PM, RichD wrote:

> He estimated music contains 40 bits/second entropy.
> How close is MP3 to that?

And .jpg sucks too. Which shows you that in spite of the years since the 
original break-throughs most people haven't a clue as to what is going 
on in this area. That does not mean, however that NOBODY understands it. 
I DO know of people (working in the telecommunications industry as 
Shannon did) who do understand this and have developed compression 
programs that actually work. Eventually I suppose they will filter down 
to us peons.

He estimated music contains 40 bits/second entropy.
How close is MP3 to that?

___________________________________
I doubt surprised Shannon said that, and if he did its somewhere between 
meaningless and wrong.

CD quality mp3s are roughly equivalent to 178 kbps, over 4,000 times his 
estimate. But then you can encode a lot of sounds that most people would not 
consider music. And it stereo, so you can halve it if Shannon was talking 
about mono.

And how do you define music, except as sound? And random sound waveforms 
cannot be compressed on average at all. To get a smaller figure for music, 
you have to define what subsets of sounds are music. Lots of luck.

The real number probably lies somewhere between 178 kbps and 40 bps. The 
actual number is the base 2 logarithm of the number of different 1 second 
sound bites that the ear can distuingish and would consider as music. I 
doubt even Shannon would have known the answer to that equation.

RichD <r_delaney2001@yahoo.com> writes:

>He estimated music contains 40 bits/second entropy.
>How close is MP3 to that?

Far away; apparently not a person who paid topdollar for gold plated
audio cables.

Casper
-- 

On Fri, 4 Nov 2011 20:33:48 +1100, "Peter Webb"
<webbfamily@optusnetDIESPAMDIE.com.au> wrote:

>
>
>
>He estimated music contains 40 bits/second entropy.
>How close is MP3 to that?
>
>___________________________________
>I doubt surprised Shannon said that, and if he did its somewhere between 
>meaningless and wrong.
>
>CD quality mp3s are roughly equivalent to 178 kbps, over 4,000 times his 
>estimate. But then you can encode a lot of sounds that most people would not 
>consider music. And it stereo, so you can halve it if Shannon was talking 
>about mono.
>
>And how do you define music, except as sound? And random sound waveforms 
>cannot be compressed on average at all. To get a smaller figure for music, 
>you have to define what subsets of sounds are music. Lots of luck.
>
>The real number probably lies somewhere between 178 kbps and 40 bps. The 
>actual number is the base 2 logarithm of the number of different 1 second 
>sound bites that the ear can distuingish and would consider as music. I 
>doubt even Shannon would have known the answer to that equation.


I expect that if Shannon said something like that, he was considering
music to be more along the lines of a MIDI stream than a digitized
sound file.

Assuming a pianist can make 20 keystroke per second (and there are
scores that require that), 40 bits is a bit light, since you need to
refine the timing a bit, and specify the force of the keystroke, as
well as deal with the releases and the pedals (although those should
need a relatively low bit rate).

But all that together would certainly be less than 1000 bits per
second.  That assumes that music is actually random keystrokes at
random times and random forces, at at very high keystroke rate - which
it is, of course, not, no more than English sentences are random
sequences of letters.

Consider that high keystroke rates pretty much require chords, and a
chord is most certainly not three or four random keys, pressed at
different times and forces - rather they're a small subset of the
possible combinations of keys within a hand-span (or two hand-spans in
some cases), hit at the same time, and with the same force (in fact
there are less than about 10,000 chords).  And just like English, you
can't really have random sequences, or you'd just have noise.

Still, 40bps intuitively feels a bit light, but not by that much. I
expect that if Shannon said something like that, he was considering
music to be more along the lines of a MIDI stream than a digitized
sound file.

Assuming a pianist can make 20 keystroke per second (and there are
scores that require that), 40 bits is a bit light, since you need to
refine the timing a bit, and specify the force of the keystroke, as
well as deal with the releases and the pedals (although those should
need a relatively low bit rate).

But all that together would certainly be less than 1000 bits per
second.  That assumes that music is actually random keystrokes at
random times and random forces, at at very high keystroke rate - which
it is, of course, not, no more than English sentences are random
sequences of letters.

Consider that high keystroke rates pretty much require chords, and a
chord is most certainly not three or four random keys, pressed at
different times and forces - rather they're a small subset of the
possible combinations of keys within a hand-span (or two hand-spans in
some cases), hit at the same time, and with the same force (in fact
there are less than about 10,000 chords).  And just like English, you
can't really have random sequences, or you'd just have noise.

Still, 40bps intuitively feels a bit light for some of the busiest
music, but not by that much.  For more typical passages, it seems
adequate.

On 04/11/2011 09:33, Peter Webb wrote:
>
>
>
> He estimated music contains 40 bits/second entropy.
> How close is MP3 to that?
>
> ___________________________________
> I doubt surprised Shannon said that, and if he did its somewhere between
> meaningless and wrong.
>

If Shannon did make the statement attributed to him above (and I 
seriously doubt it) - I would like to see references.

> CD quality mp3s are roughly equivalent to 178 kbps, over 4,000 times his
> estimate. But then you can encode a lot of sounds that most people would
> not consider music. And it stereo, so you can halve it if Shannon was
> talking about mono.
>
> And how do you define music, except as sound? And random sound waveforms


Have you never seen sheet music? That is what Shannon was estimating - 
the bitrate for describing music in the abstract. There are a finite 
number of notes, durations and amplitudes in a classical composition.

I suspect 40 bits/sec is still far too tight, but a midi stream using a 
high end reconstruction codec represents a pretty good example of what 
is possible by way of compression for *music* as opposed to voice or a 
random noise stream.

> cannot be compressed on average at all. To get a smaller figure for
> music, you have to define what subsets of sounds are music. Lots of luck.

I think that may have been his intention although I don't actually 
recall seeing the 40 bit/s number originally attributed to him above.

As I said if he did anything I think he was estimating the information 
content of music in the already concise form of an orchestral score.
>
> The real number probably lies somewhere between 178 kbps and 40 bps. The
> actual number is the base 2 logarithm of the number of different 1
> second sound bites that the ear can distuingish and would consider as
> music. I doubt even Shannon would have known the answer to that equation.

Maybe he didn't consider all the alternative atonal and continuous 
frequency synthesisers of arbitrary waveforms that are possible now - or 
perhaps it was a subtle dig at Stockhausen and Schoenberg as not music.

Be interesting if the reference to this paper can be found to see how he 
allocated those 40 bits/s....

I reckon at a bare minimum about 7 to the note, 8 to amplitude, 6 
duration, 5 to the instrument - and it is already obvious that you 
cannot encode more than a single note per second at this bitrate.

Can anyone provide a citation to this alleged paper on music bitrate?

-- 
Regards,
Martin Brown