information theory?

In sci.math Robert Wessel <robertwessel2@yahoo.com> wrote:
...
> Consider that high keystroke rates pretty much require chords, and a
> chord is most certainly not three or four random keys, pressed at
> different times and forces - rather they're a small subset of the
> possible combinations of keys within a hand-span (or two hand-spans in
> some cases), hit at the same time, and with the same force (in fact
> there are less than about 10,000 chords).  And just like English, you
> can't really have random sequences, or you'd just have noise.
> 
> Still, 40bps intuitively feels a bit light, but not by that much. I
> expect that if Shannon said something like that, he was considering
> music to be more along the lines of a MIDI stream than a digitized
> sound file.
...

Perhaps we could use spoken language as a guide to the auditory bidrate
we've evolved to process. As Shannon estimated, each letter in English
has only a couple bits of entropy, with normal reading speed (that
uses the auditory processing -- the reason you can't count the number
of words in a limerick without using your fingers) around 150 wpm.
Let's call that 25 bps.

40 bps is then about the "square" (in log terms :) of our evolved language
processing speeds. 

Maybe not such a bad estimate.

The same disparity is then seen between normal good voice compression
and the actually bitrate we operate on. I.e. 25 bps "back-end" processing 
speed, yet voice compression being still 1000s of bps.

In some applications  over the years I've used speach-to-text and text-to-speach
over very low speed lines as a slap-dash compression method. With a bit
of a phoneme "nomenclator" it wasn't too bad. No doubt the idea is still in
use, but I don't think I've heard of it in preference to straight dsp methods.

-- 
co2 has no climate forcing effect and is not a greenhouse gas and, for that 
matter, neither is water vapour. 
-- BONZO@27-32-240-172 [100s of nyms], 5 Sep 2009

Earth's atmosphere contains natural greenhouse gases (mostly water
vapor, carbon dioxide, and methane) which act to keep the lower layers
of the atmosphere warmer than they otherwise would be without those gases. 
-- Dr Roy W. Spencer, "Global Warming", 2008

This is what the real climate scientist Dr Roy Spencer said.
-- BONZO@27-32-240-172 [daily nymshifter], 3 Mar 2011 16:29 +1100

"Martin Brown" <|||newspam|||@nezumi.demon.co.uk> wrote in message 
news:I9Psq.13644$Mg.12196@newsfe13.iad...
> On 04/11/2011 09:33, Peter Webb wrote:
>>
>>
>>
>> He estimated music contains 40 bits/second entropy.
>> How close is MP3 to that?
>>
>> ___________________________________
>> I doubt surprised Shannon said that, and if he did its somewhere between
>> meaningless and wrong.
>>
>
> If Shannon did make the statement attributed to him above (and I seriously 
> doubt it) - I would like to see references.
>
>> CD quality mp3s are roughly equivalent to 178 kbps, over 4,000 times his
>> estimate. But then you can encode a lot of sounds that most people would
>> not consider music. And it stereo, so you can halve it if Shannon was
>> talking about mono.
>>
>> And how do you define music, except as sound? And random sound waveforms
>
>
> Have you never seen sheet music? That is what Shannon was estimating - the 
> bitrate for describing music in the abstract. There are a finite number of 
> notes, durations and amplitudes in a classical composition.
>

Far more plausible.

If you don't consider singing to be music.


> I suspect 40 bits/sec is still far too tight, but a midi stream using a 
> high end reconstruction codec represents a pretty good example of what is 
> possible by way of compression for *music* as opposed to voice or a random 
> noise stream.
>

I wonder what Jimi Hendrix playing Star Spangled Banner would sound like as 
a midi stream?



>> cannot be compressed on average at all. To get a smaller figure for
>> music, you have to define what subsets of sounds are music. Lots of luck.
>
> I think that may have been his intention although I don't actually recall 
> seeing the 40 bit/s number originally attributed to him above.
>
> As I said if he did anything I think he was estimating the information 
> content of music in the already concise form of an orchestral score.

= midi. A very plausible explanation, but one which requires a very limitted 
definition of music.


>>
>> The real number probably lies somewhere between 178 kbps and 40 bps. The
>> actual number is the base 2 logarithm of the number of different 1
>> second sound bites that the ear can distuingish and would consider as
>> music. I doubt even Shannon would have known the answer to that equation.
>
> Maybe he didn't consider all the alternative atonal and continuous 
> frequency synthesisers of arbitrary waveforms that are possible now - or 
> perhaps it was a subtle dig at Stockhausen and Schoenberg as not music.
>

Yeah, I was going to use a synthesiser as an example of an instrument that 
couldn't be compressed, but then I realised that Shannon didn't have to 
worry about digital synthesisers in his day.


> Be interesting if the reference to this paper can be found to see how he 
> allocated those 40 bits/s....
>
> I reckon at a bare minimum about 7 to the note, 8 to amplitude, 6 
> duration, 5 to the instrument - and it is already obvious that you cannot 
> encode more than a single note per second at this bitrate.
>
> Can anyone provide a citation to this alleged paper on music bitrate?
>
> -- 
> Regards,
> Martin Brown 

"Peter Webb" <webbfamily@optusnetDIESPAMDIE.com.au> wrote in message 
news:4eb3b1c6$0$13394$afc38c87@news.optusnet.com.au...
|
|
|
| He estimated music contains 40 bits/second entropy.
| How close is MP3 to that?
|
| ___________________________________
| I doubt surprised Shannon said that, and if he did its somewhere between
| meaningless and wrong.
|
| CD quality mp3s are roughly equivalent to 178 kbps, over 4,000 times his
| estimate. But then you can encode a lot of sounds that most people would 
not
| consider music. And it stereo, so you can halve it if Shannon was talking
| about mono.

Stereo is multiplexed. When the left channel is played the right channel is
MISSING, and vice versa. One of the earliest compression algorithms was
run length encoding, whereby a byte or short string of bytes is transmitted
as one byte and a count for the number of times the byte is to be repeated.
When the doctor tells you to say "ahhhhhh....." then that encodes to 'a'
once, 'h' 6 times, '.' 5 times.  "a1h6.5".
Text (word) compression is very easy, construct 16 lists of words
with that number of letters, up to 256 words per list. That gives a
vocabulary of up to 4096 words. Now encode each word by its number
of letters and its entry into its list using 2 bytes, and you still have 4 
bits
left over for punctuation. For example, the word "doctor" is to be found
in the list of 6 letter words and might be the 32nd entry, so it is encoded
as 06:32, two bytes instead of seven, the trailing space can be assumed
unless modified by the punctuation nybble.

<kym@kymhorsell.com> wrote in message 
news:4eb3c38a$0$22471$afc38c87@news.optusnet.com.au...
> In sci.math Robert Wessel <robertwessel2@yahoo.com> wrote:
> ...
>> Consider that high keystroke rates pretty much require chords, and a
>> chord is most certainly not three or four random keys, pressed at
>> different times and forces - rather they're a small subset of the
>> possible combinations of keys within a hand-span (or two hand-spans in
>> some cases), hit at the same time, and with the same force (in fact
>> there are less than about 10,000 chords).  And just like English, you
>> can't really have random sequences, or you'd just have noise.
>>
>> Still, 40bps intuitively feels a bit light, but not by that much. I
>> expect that if Shannon said something like that, he was considering
>> music to be more along the lines of a MIDI stream than a digitized
>> sound file.
> ...
>
> Perhaps we could use spoken language as a guide to the auditory bidrate
> we've evolved to process. As Shannon estimated, each letter in English
> has only a couple bits of entropy, with normal reading speed (that
> uses the auditory processing -- the reason you can't count the number
> of words in a limerick without using your fingers) around 150 wpm.
> Let's call that 25 bps.
>
> 40 bps is then about the "square" (in log terms :) of our evolved language
> processing speeds.
>
> Maybe not such a bad estimate.
>
> The same disparity is then seen between normal good voice compression
> and the actually bitrate we operate on. I.e. 25 bps "back-end" processing
> speed, yet voice compression being still 1000s of bps.
>
> In some applications  over the years I've used speach-to-text and 
> text-to-speach
> over very low speed lines as a slap-dash compression method. With a bit
> of a phoneme "nomenclator" it wasn't too bad. No doubt the idea is still 
> in
> use, but I don't think I've heard of it in preference to straight dsp 
> methods.
>

I had a play with a TI phoneme generator chip hooked up to a 6502 
microproceesor in 1979 or 80. Couldn't get a decent "fuck" out of it - came 
out as a flat "fut". Times have changed.



> -- 
> co2 has no climate forcing effect and is not a greenhouse gas and, for 
> that
> matter, neither is water vapour.
> -- BONZO@27-32-240-172 [100s of nyms], 5 Sep 2009
>
> Earth's atmosphere contains natural greenhouse gases (mostly water
> vapor, carbon dioxide, and methane) which act to keep the lower layers
> of the atmosphere warmer than they otherwise would be without those gases.
> -- Dr Roy W. Spencer, "Global Warming", 2008
>
> This is what the real climate scientist Dr Roy Spencer said.
> -- BONZO@27-32-240-172 [daily nymshifter], 3 Mar 2011 16:29 +1100 

"Robert Wessel" <robertwessel2@yahoo.com> wrote in message 
news:r6d7b7l6nt23kcu3n0cisoe85j0d6go5no@4ax.com...
| On Fri, 4 Nov 2011 20:33:48 +1100, "Peter Webb"
| <webbfamily@optusnetDIESPAMDIE.com.au> wrote:
|
| >
| >
| >
| >He estimated music contains 40 bits/second entropy.
| >How close is MP3 to that?
| >
| >___________________________________
| >I doubt surprised Shannon said that, and if he did its somewhere between
| >meaningless and wrong.
| >
| >CD quality mp3s are roughly equivalent to 178 kbps, over 4,000 times his
| >estimate. But then you can encode a lot of sounds that most people would 
not
| >consider music. And it stereo, so you can halve it if Shannon was talking
| >about mono.
| >
| >And how do you define music, except as sound? And random sound waveforms
| >cannot be compressed on average at all. To get a smaller figure for 
music,
| >you have to define what subsets of sounds are music. Lots of luck.
| >
| >The real number probably lies somewhere between 178 kbps and 40 bps. The
| >actual number is the base 2 logarithm of the number of different 1 second
| >sound bites that the ear can distuingish and would consider as music. I
| >doubt even Shannon would have known the answer to that equation.
|
|
| I expect that if Shannon said something like that, he was considering
| music to be more along the lines of a MIDI stream than a digitized
| sound file.
|
| Assuming a pianist can make 20 keystroke per second (and there are
| scores that require that), 40 bits is a bit light, since you need to
| refine the timing a bit, and specify the force of the keystroke, as
| well as deal with the releases and the pedals (although those should
| need a relatively low bit rate).
|
| But all that together would certainly be less than 1000 bits per
| second.  That assumes that music is actually random keystrokes at
| random times and random forces, at at very high keystroke rate - which
| it is, of course, not, no more than English sentences are random
| sequences of letters.
|
| Consider that high keystroke rates pretty much require chords, and a
| chord is most certainly not three or four random keys, pressed at
| different times and forces - rather they're a small subset of the
| possible combinations of keys within a hand-span (or two hand-spans in
| some cases), hit at the same time, and with the same force (in fact
| there are less than about 10,000 chords).  And just like English, you
| can't really have random sequences, or you'd just have noise.
|
| Still, 40bps intuitively feels a bit light, but not by that much. I
| expect that if Shannon said something like that, he was considering
| music to be more along the lines of a MIDI stream than a digitized
| sound file.
|
| Assuming a pianist can make 20 keystroke per second (and there are
| scores that require that), 40 bits is a bit light, since you need to
| refine the timing a bit, and specify the force of the keystroke, as
| well as deal with the releases and the pedals (although those should
| need a relatively low bit rate).
|
| But all that together would certainly be less than 1000 bits per
| second.  That assumes that music is actually random keystrokes at
| random times and random forces, at at very high keystroke rate - which
| it is, of course, not, no more than English sentences are random
| sequences of letters.
|
| Consider that high keystroke rates pretty much require chords, and a
| chord is most certainly not three or four random keys, pressed at
| different times and forces - rather they're a small subset of the
| possible combinations of keys within a hand-span (or two hand-spans in
| some cases), hit at the same time, and with the same force (in fact
| there are less than about 10,000 chords).  And just like English, you
| can't really have random sequences, or you'd just have noise.
|
| Still, 40bps intuitively feels a bit light for some of the busiest
| music, but not by that much.  For more typical passages, it seems
| adequate.

The difference between MIDI and audio compression is the
synthesizer. The same pianist can play 20 notes a second on
a church organ or harpsichord and it will sound bloody awful,
the pipes don't have the same percussive attack that a hammer
on a string has. Same with speech, it is the explosive attack of
the consonant that makes the word, not the drawn out vowel
which is often changed by accent. Orz-dry-lee-uns still write
"Australians" but change the sound of the vowels as we all do.
Much of the spoken word is clich&#4294967295;, "it's like" "know what I
mean".  The first four notes of Beethoven's 5th are a clich&#4294967295;,
it wouldn't matter what instrument they were played on, you'd
still recognise the phrase. Audio compression carries more
information than MIDI, it has to.

"Peter Webb" <webbfamily@optusnetDIESPAMDIE.com.au> writes:

> I had a play with a TI phoneme generator chip hooked up to a 6502 
> microproceesor in 1979 or 80. Couldn't get a decent "fuck" out of it - came 
> out as a flat "fut". Times have changed.

Well, technology has changed, but your choice of test words shows human
nature hasn't.   
-- 
Jesse F. Hughes

"C is for Cookie.  That's good enough for me."  
                                     Cookie Monster

On Fri, 04 Nov 2011 09:13:31 -0400, "Jesse F. Hughes"
<jesse@phiwumbda.org> wrote:

>"Peter Webb" <webbfamily@optusnetDIESPAMDIE.com.au> writes:
>
>> I had a play with a TI phoneme generator chip hooked up to a 6502 
>> microproceesor in 1979 or 80. Couldn't get a decent "fuck" out of it - came 
>> out as a flat "fut". Times have changed.
>
>Well, technology has changed, but your choice of test words shows human
>nature hasn't.   

Reminds me of this painting...
http://www.barber.org.uk/653.html

Lots of things have changed in ~300 years, but boys have not. 

kym@kymhorsell.com wrote:
> In sci.math Robert Wessel <robertwessel2@yahoo.com> wrote:
> ...
>> Consider that high keystroke rates pretty much require chords, and a
>> chord is most certainly not three or four random keys, pressed at
>> different times and forces - rather they're a small subset of the
>> possible combinations of keys within a hand-span (or two hand-spans in
>> some cases), hit at the same time, and with the same force (in fact
>> there are less than about 10,000 chords).  And just like English, you
>> can't really have random sequences, or you'd just have noise.
>>
>> Still, 40bps intuitively feels a bit light, but not by that much. I
>> expect that if Shannon said something like that, he was considering
>> music to be more along the lines of a MIDI stream than a digitized
>> sound file.
> ...
> 
> Perhaps we could use spoken language as a guide to the auditory bidrate
> we've evolved to process. As Shannon estimated, each letter in English
> has only a couple bits of entropy, with normal reading speed (that
> uses the auditory processing -- the reason you can't count the number
> of words in a limerick without using your fingers) around 150 wpm.
> Let's call that 25 bps.
> 
> 40 bps is then about the "square" (in log terms :) of our evolved language
> processing speeds. 
> 
> Maybe not such a bad estimate.
> 
> The same disparity is then seen between normal good voice compression
> and the actually bitrate we operate on. I.e. 25 bps "back-end" processing 
> speed, yet voice compression being still 1000s of bps.
> 
> In some applications  over the years I've used speach-to-text and text-to-speach
> over very low speed lines as a slap-dash compression method. With a bit
> of a phoneme "nomenclator" it wasn't too bad. No doubt the idea is still in
> use, but I don't think I've heard of it in preference to straight dsp methods.
> 
At the 1939 World Fair in New York City, Bell Telephone had an exhibit 
featuring the "vocoder".  An operator could synchronize speech by 
pressing "keys" on a console.  Bell thought the method could be used for 
"compressing" its telephone bandwidth requirements.

As it turned out, bandwidth was not the limiting factor in future 
telephone growth.  Simple system changes like separating the low-speed 
"housekeeping" functions from the in-band "information" were implemented.

I was involved in the testing of microwave frequencies in small circular 
wave-guide to allow for much wider band channels.  The need never 
evolved, and fiber optics replaced microwaves where it was needed.

As far as I know, any "compression" system requires some means of 
"storing" information so that it can be processed.  This was just being 
developed during Shannon's time.  It's obvious that "digital" storage is 
easier and more flexible than analog.

-- 
Virg Wall

"Spehro Pefhany" <speffSNIP@interlogDOTyou.knowwhat> wrote in message 
news:6ot7b7d7pkgh2c80oquvrd14chprebkob1@4ax.com...
| On Fri, 04 Nov 2011 09:13:31 -0400, "Jesse F. Hughes"
| <jesse@phiwumbda.org> wrote:
|
| >"Peter Webb" <webbfamily@optusnetDIESPAMDIE.com.au> writes:
| >
| >> I had a play with a TI phoneme generator chip hooked up to a 6502
| >> microproceesor in 1979 or 80. Couldn't get a decent "fuck" out of it - 
came
| >> out as a flat "fut". Times have changed.
| >
| >Well, technology has changed, but your choice of test words shows human
| >nature hasn't.
|
| Reminds me of this painting...
| http://www.barber.org.uk/653.html
|
| Lots of things have changed in ~300 years, but boys have not.
|
Sex is 3 billion years old and still it fascinates the young who imagine
they invented its variations.

["Followup-To:" header set to sci.crypt.]
On 2011-11-04, Casper H.S  Dik <Casper.Dik@OrSPaMcle.COM> wrote:
> RichD <r_delaney2001@yahoo.com> writes:
>
>>He estimated music contains 40 bits/second entropy.
>>How close is MP3 to that?
>
> Far away; apparently not a person who paid topdollar for gold plated
> audio cables.

Even had they existed, he wouldn't have. He was an engineer, for whom
looks and cost did not determine function. 

>
> Casper