information theory?

On Nov 2, 12:02&#4294967295;pm, Martin Brown <|||newspam...@nezumi.demon.co.uk>
wrote:
> On 02/11/2011 09:48, JohnF wrote:
> > Just curious -- if it was freely available online at one time,
> > and I assume legally at that time, how does later publishing
> > a paper copy retroactively affect online legality? I'd have
> > assumed any originally-legal online info remains legal regardless
> > of who else prints it on paper. Is that wrong?

It doesn't - not automatically anyway. Note that you also don't
automatically have the right to redistribute a work, even if it was
previously available online or via carrier pigeon. Your (!) licence
has to grant you that right. OTOH, *use* of a work (as in, reading it)
is not reserved for the copyright holder. So if you already have a
copy, presumably legally obtained, you can continue reading it.

> I don't know. ISTR it was online as a part of a global proofing effort.
> When that was finished it looks like the online copy was removed.
>
> (at least if it is still there I can't see it now)
>
> It was at:http://bayes.wustl.edu/etj/prob.html

Try http://web.archive.org/web/20020124204123/http://bayes.wustl.edu/etj/prob.html

For extra credit: discuss what rights the wayback machine has or
presumes in order to meet its mission.

BJACOBY@teranews.com wrote:
> On 11/1/2011 12:29 PM, RichD wrote:
>
>> I read Shannon's paper long go, and found it fascinating
>> ever since. In the most general sense, it's applicable
>> to so many areas. For instance, people have recast
>> thermodynamics as information processing. Or, you
>> could model the entire universe as a computer, and
>> information theory applies. It even describes human
>> nature - manipulating symbols is what separates us
>> from the apes.
>
> Actually, Shannon's "paper" is a book. And the unfortunate part is that
> physicists in particular for the most part have developed little
> understanding of Shannon's association of information with entropy. This
> one fact alone blows a huge hole in classical physics. Yet, (nearly) all
> physicists go blindly on barfing the sad dogma of how entropy always
> increases, how the universe is "running down" etc. etc. They know even
> less about the relationships between signal processing and
> Electromagnetic theory. They can't understand the transform
> relationships between the amplitude of a wave in free space at one point
> and it's far field angular expansion. They are snowed by self-inductance
> because they haven't bothered to understand the nature of feedback
> systems. In short they are stuck in ignorance and the past and have NO
> desire to move out of it.
>
> But the unfortunate facts are that information theory especially
> combined with things like classical thermodynamics holds the key to
> physics in the 21st century! And what exactly IS physics going to be in
> the 21st century? I can tell you. It will be the physics of the things
> that physics of today has so LOUDLY avoided. Namely, the physics of
> life! The grass under my brick sidewalk is ALWAYS striving to take the
> chaos of the dirt down there and turn it into ORDER! It insists on doing
> this every year in spite of my best efforts to stop it! But still
> "traditional" science pretends to move "forward" while ignoring the fact
> that Shannon shot all their dogma to hell back in the MIDDLE of the the
> 20th century.
>


Eh? Physics guys loudly accept that entropy can go up locally. They
just  say universal ( meaning at the scale of the universe ) entropy
is what goes up. Sagan at one point assigned values to the relative 
entropy/order increases from reading something, for example.

Penrose/Hawking and everybody else since have used
communications theory quite a bit to do physics. I
don't even think it's settled yet.

Penroses' "Emperor's New Mind" misses quite a bit ( he
skyhooks some stuff ) but that alone should show they're
thinking about it.

If physics has a problem, it is that the experiments have all gotten
too expensive for a 1920s-1950s level of progress to occur. That, and
string  theory becoming too precious and abstract.

--
Les Cargill

RichD wrote:
> On Nov 2, Chris <chris.sant...@gmail.com> wrote:
>> Be careful going down this path. For me it was
>> much more interesting in theory than in practice.
>>
>> Me in 2008: "Like, what *is* the nature of information,
>> man?" (waves hand to clear smoke from the room)
>>
>> Then I signed up for Info Theory, and I spent
>> a whole semester taking integrals of logs of
>> probability distributions and proving that one
>> was bounded by the other.
> 
> It's too bad you got so little out of it.
> It's not 'just a theory', it's a field
> where theory and practice coincide.
> The theorists concoct the recipes, the
> engineers follow the instructions, and
> it works, bang on.  Fukengrooven.
> And it's the root of many a Silly Con
> Valley fortune.
> 
> I strongly suggest, if you have some
> free neurons and time and nothing's good on
> tv, read Shannon's original paper.  It's
> utterly fucking brilliant.  There's no doubt,
> in Valhalla, Shannon shares a table with Einstein.
> 
I was at Bell Labs in 1949, and Betty Moore, Claude Shannon's wife, 
worked in the same department.  She complained that their new carpet 
didn't have deep enough nap to hide the solder droppings from the 
computer Claude was building.  It could perform math functions with its 
input/output being in Roman numerals.

> It's sort of funny, he called it communication
> theory, it was his disciples who popularized
> 'information theory'.
> 
Shannon took a very practical outlook on "information".  I recall how 
he would stop people in the halls and ask them to guess the next letter 
in a sentence he'd show them.  I remember one as: "A motorcycle has no 
reverse; it can not back up."  He used this method to determine the 
redundancy in the English language.

Most of the time, he was riding his unicycle down the hall!

-- 
Virg Wall

In article <E8udnR4LarelIizTnZ2dnUVZ_ridnZ2d@earthlink.com>,
 VWWall <vwall@large.invalid> wrote:

[....]

> I was at Bell Labs in 1949, and Betty Moore, Claude Shannon's wife, 
> worked in the same department.  She complained that their new carpet 
> didn't have deep enough nap to hide the solder droppings from the 
> computer Claude was building.  It could perform math functions with its 
> input/output being in Roman numerals.
> 
> > It's sort of funny, he called it communication
> > theory, it was his disciples who popularized
> > 'information theory'.
> > 
> Shannon took a very practical outlook on "information".  I recall how 
> he would stop people in the halls and ask them to guess the next letter 
> in a sentence he'd show them.  I remember one as: "A motorcycle has no 
> reverse; it can not back up."  He used this method to determine the 
> redundancy in the English language.
> 
> Most of the time, he was riding his unicycle down the hall!

The seminal introductory work for me was the Undergraduate text
"Coding and Information Theory", by the late Richard W. Hamming,
1980, ISBN 0-13-139139-9 (perhaps one of the 1st books to have
an ISBN number). &#4294967295;It cost me $26.35 (CDN), hardcover, and I
devoured every morsel of it.

His later work "The Art of Probability" is also worth a read for
anyone who still believes in randomness.

outer

--&#4294967295;
Random : An infinitesimal, yet omnidimensional, god of science.
Random often appears in the guise of a trickster named Error.

>BJACOBY@teranews.com wrote:
>> On 11/1/2011 12:29 PM, RichD wrote:
>>
>>> I read Shannon's paper long go, and found it fascinating
>>> ever since. In the most general sense, it's applicable
>>> to so many areas. For instance, people have recast
>>> thermodynamics as information processing. Or, you
>>> could model the entire universe as a computer, and
>>> information theory applies. It even describes human
>>> nature - manipulating symbols is what separates us
>>> from the apes.
>>
>> Actually, Shannon's "paper" is a book. And the unfortunate part is that
>> physicists in particular for the most part have developed little
>> understanding of Shannon's association of information with entropy.
This
>> one fact alone blows a huge hole in classical physics. Yet, (nearly)
all
>> physicists go blindly on barfing the sad dogma of how entropy always
>> increases, how the universe is "running down" etc. etc. They know even
>> less about the relationships between signal processing and
>> Electromagnetic theory. They can't understand the transform
>> relationships between the amplitude of a wave in free space at one
point
>> and it's far field angular expansion. They are snowed by
self-inductance
>> because they haven't bothered to understand the nature of feedback
>> systems. In short they are stuck in ignorance and the past and have NO
>> desire to move out of it.
>>
>> But the unfortunate facts are that information theory especially
>> combined with things like classical thermodynamics holds the key to
>> physics in the 21st century! And what exactly IS physics going to be in
>> the 21st century? I can tell you. It will be the physics of the things
>> that physics of today has so LOUDLY avoided. Namely, the physics of
>> life! The grass under my brick sidewalk is ALWAYS striving to take the
>> chaos of the dirt down there and turn it into ORDER! It insists on
doing
>> this every year in spite of my best efforts to stop it! But still
>> "traditional" science pretends to move "forward" while ignoring the
fact
>> that Shannon shot all their dogma to hell back in the MIDDLE of the the
>> 20th century.
>>
>
>
>Eh? Physics guys loudly accept that entropy can go up locally. They
>just  say universal ( meaning at the scale of the universe ) entropy
>is what goes up. Sagan at one point assigned values to the relative 
>entropy/order increases from reading something, for example.
>
>Penrose/Hawking and everybody else since have used
>communications theory quite a bit to do physics. I
>don't even think it's settled yet.
>
>Penroses' "Emperor's New Mind" misses quite a bit ( he
>skyhooks some stuff ) but that alone should show they're
>thinking about it.
>
>If physics has a problem, it is that the experiments have all gotten
>too expensive for a 1920s-1950s level of progress to occur. That, and
>string  theory becoming too precious and abstract.

In basic high school physics and chemistry, when you first learn about
entropy, you learn about it rising and falling through various processes.
Its there where you learn the, perhaps, surprising result that the *net*
entropy along any chain of events always increases. Why would even someone
limited to high school science find anything anomalous about localised
entropy in information theory?

Steve

On Tue, 01 Nov 2011 22:38:03 -0500, BJACOBY@teranews.com wrote:

> On 11/1/2011 12:29 PM, RichD wrote:
> 
>> I read Shannon's paper long go, and found it fascinating ever since. 
>> In the most general sense, it's applicable to so many areas.  For
>> instance, people have recast thermodynamics as information processing. 
>> Or, you could model the entire universe as a computer, and information
>> theory applies.  It even describes human nature - manipulating symbols
>> is what separates us from the apes.
> 
> Actually, Shannon's "paper" is a book. And the unfortunate part is that
> physicists in particular for the most part have developed little
> understanding of Shannon's association of information with entropy. This
> one fact alone blows a huge hole in classical physics. Yet, (nearly) all
> physicists go blindly on barfing the sad dogma of how entropy always
> increases, how the universe is "running down" etc. etc. They know even
> less about the relationships between signal processing and
> Electromagnetic theory. They can't understand the transform
> relationships between the amplitude of a wave in free space at one point
> and it's far field angular expansion. They are snowed by self-inductance
> because they haven't bothered to understand the nature of feedback
> systems. In short they are stuck in ignorance and the past and have NO
> desire to move out of it.
> 
> But the unfortunate facts are that information theory especially
> combined with things like classical thermodynamics holds the key to
> physics in the 21st century! And what exactly IS physics going to be in
> the 21st century? I can tell you. It will be the physics of the things
> that physics of today has so LOUDLY avoided. Namely, the physics of
> life! The grass under my brick sidewalk is ALWAYS striving to take the
> chaos of the dirt down there and turn it into ORDER! It insists on doing
> this every year in spite of my best efforts to stop it! But still
> "traditional" science pretends to move "forward" while ignoring the fact
> that Shannon shot all their dogma to hell back in the MIDDLE of the the
> 20th century.

This almost makes sense... at least at first.

But the earliest I can remember someone doing otherwise, in earnest, is 
1977, with the publication of Kantor's "Information Mechanics".  There 
are probably earlier examples.

-- 
RLW

In article <DP-dnaStmoSaeSzTnZ2dnUVZ5oadnZ2d@giganews.com>, Lofty Goat
<rlwatkins@gmail.com> wrote:

> 
> But the earliest I can remember someone doing otherwise, in earnest, is 
> 1977, with the publication of Kantor's "Information Mechanics".  There 
> are probably earlier examples.
I am delighted to learn that somebody besides me knows about that book.
What do you think of it?

-- 
Chris Henrich
http://www.mathinteract.com
God just doesn't fit inside a single religion.

On 11/2/2011 1:51 PM, RichD wrote:

> I strongly suggest, if you have some
> free neurons and time and nothing's good on
> tv, read Shannon's original paper.  It's
> utterly fucking brilliant.  There's no doubt,
> in Valhalla, Shannon shares a table with Einstein.
>
> It's sort of funny, he called it communication
> theory, it was his disciples who popularized
> 'information theory'.

Not funny at all. He worked for Bell Telephone. Duh! And indeed the 
original work DID have the major thrust of a theory of communication. 
Later the breakthrough nature of his ideas were recognized as having 
much wider general application in MANY at first seemingly unrelated 
fields. Hence the name with a wider swath: Information theory.

And yes Shannon's ideas on information including those on Cryptography 
(some probably still classified) definitely ARE "utterly fucking 
brilliant"!

On 02/11/2011 18:38, RichD wrote:
> On Nov 2, Martin Brown<|||newspam...@nezumi.demon.co.uk>  wrote:
>>>> Do you have a specific question? Or just want
>>>> to lurk and learn?
>>
>>> I'd like to lurk someplace, where there are
>>> discussions of current topics.  It's such a
>>> rich field, there are always new ideas.
>>
>>> I read Shannon's paper long go, and found it fascinating
>>> ever since.  In the most general sense, it's
>>> applicable to so many areas.
>>
>>>>>> Are there any boards, or private blogs, dedicated
>>>>>> to discussion of information theory?
>>
>> You don't really say which part of information
>> theory you are interested in -
>
> Coding, antenna arrays, compression. etc.

You will have to look hard and get invited to join any of the private 
lists that do exist. Like most of the academic topics and some other 
research lists into artificial intelligence they did once have Usenet 
groups but because of nutters and hipcryme flooding have moved to 
mailing lists / private blogs or other externally invisible means.
>
>> You could do worse than look for the collected works
>> of Ed Jaynes -
>
> He's the one who published the original work
> re-interpreting thermodynamics as information theory.
>
> It was ingenious, but made no new predictions,
> so the physics community yawned and said,
> what's the point?

That is an extremely unfair caricature. His contribution to information 
theory as applied to Bayesian inference and improper priors was widely 
respected in the experimental physics community and grudgingly accepted 
as a worthwhile challenge to the status quo by hard line frequentalists 
in the pure mathematics department. His maximum entropy based analysis 
of the bias in Wolfs dice data in the paper "Where do we stand on 
Maximum Entropy" Brandeis Lecture series was seminal.

The big difference now is that we have enough computing power to do a 
full Bayesian analysis these days rather than an approximation.
>
>> it used to be available online at one time but
>> now it is published.
>>
>> http://www.amazon.com/T-Jaynes-Probability-Statistics-Statistical/dp/...
>
> I'm not looking for textbooks, but a place
> to hang out with experts, discussing the
> latest stuff.  Like the faculty lounge -

They will find you if you are interesting enough.

-- 
Regards,
Martin Brown

In sci.crypt Les Cargill <lcargill99@comcast.com> wrote:
> 
> Penroses' "Emperor's New Mind" misses quite a bit ( he skyhooks some
> stuff ) but that alone should show they're thinking about it.

_The Road to Reality_ goes into a lot of detail.

Andrew.