John E. Hadstate wrote:>>>They can't be quite the same thing since thermodynamic >>>entropy measures a property of physical systems while >>>information theoretic entropy measures a property of >>>something that is a figment of our imaginations (a useful >>>figment, to be sure, but "bits" have no physical >>>manifestation and are not constrained by the laws of >>>physics). >> >> Claude Shannon didn't pick the term 'entropy' by >>coincidence. Thermodynamic entropy and cryptographic >>entropy are both the same function of the probabilities of >>a system being in each of its possible states. > > What does it mean for a continuous, physical system to be > "in each of its possible states"? So far as I know,I don't know where this discussion came from (my browser only shows this and the immediately previous post), but from what I remember from my Physics course, the "intuitive" notion of entropy (in Thermodynamics) was something like a measure of "disorder", a measure of "chaos/disaster" (I hope the term chaos doesn't get mixed up with some other mathematical theory -- if so, please do not go off that tangent just because I misused the term :-)). In that sense, the I.T. notion of entropy is closely related to that -- a soruce that has a lot of order and clearly defined patterns has low entropy. A very unpredictable, chaotic/disorganized source of events has a lot of entropy... Makes perfect sense to me (from the intuitive point of view). More formally, if the entropy is defined for the discrete case as the average self-information, the notion can be easily extended to the continuous case by replacing the sum by an integral, and the probability by probability density!! I don't see why that would be a problem; the notion of what's happening is the same: you get maximum entropy for a flat probability distribution, etc. etc. > thermodynamics pre-dates quantum theory and, as such, views > the physical world as a continuum. I don't believe this is correct -- well, I should rather say that Quantum theory has nothing to do with it; and certainly thermodynamics does not view the physical world as a continuum -- the notion of *molecules* and *atoms* certainly has to pre-date Thermodynamics, even if the classical view of atoms was different than the post-Quantum Physics view. Carlos --
Online · Live + on demand
February 9-11, 2027
Price goes up Nov 1: register now to lock in today's rate
Register Now →
31 bit pseudo-random number gen in C, C++ & dsPIC assembly code
Started by ●September 27, 2005
Reply by ●October 8, 20052005-10-08
Reply by ●October 8, 20052005-10-08
John E. Hadstate wrote:> "Mike Amling" <nospam@nospam.com> wrote in message > news:pOW1f.476$Ql1.53@chiapp18.algx.net... > >>John E. Hadstate wrote: >> >>>>It's not an unfortunate name collision, they're the same >>>>thing (if possibly scaled differently). There's a >>>>Richard Feynman book on the thermodynamics of computation >>>>that explains this, and other concepts that seem really >>>>twisted but really aren't after you think hard enough >>>>about them*. >>> >>> >>>They can't be quite the same thing since thermodynamic >>>entropy measures a property of physical systems while >>>information theoretic entropy measures a property of >>>something that is a figment of our imaginations (a useful >>>figment, to be sure, but "bits" have no physical >>>manifestation and are not constrained by the laws of >>>physics). >> >> Claude Shannon didn't pick the term 'entropy' by >>coincidence. Thermodynamic entropy and cryptographic >>entropy are both the same function of the probabilities of >>a system being in each of its possible states. >> > > > What does it mean for a continuous, physical system to be > "in each of its possible states"? So far as I know, > thermodynamics pre-dates quantum theory and, as such, views > the physical world as a continuum.The thermodynamics of gases is predicated on their consisting of discrete molecules. What thermodynamics describes is their aggregate statistical properties. Jerry -- Engineering is the art of making what you want from things you can get. �����������������������������������������������������������������������
Reply by ●October 9, 20052005-10-09
Carlos Moreno <moreno_at_mochima_dot_com@xx.xxx> writes:>John E. Hadstate wrote:>>>>information theoretic entropy measures a property of >>>>something that is a figment of our imaginations (a useful >>>>figment, to be sure, but "bits" have no physical >>>>manifestation and are not constrained by the laws of >>>>physics). >>> >>> Claude Shannon didn't pick the term 'entropy' by >>>coincidence. Thermodynamic entropy and cryptographic >>>entropy are both the same function of the probabilities of >>>a system being in each of its possible states. >> >> What does it mean for a continuous, physical system to be >> "in each of its possible states"? So far as I know,...> > thermodynamics pre-dates quantum theory and, as such, views > > the physical world as a continuum.Thermodynamics simply postulates entropy. Stat mech tries to explain it. And one of the real problems in classical physics was precisely the continuity of classical physics and the definition of entropy. Quantum theor relieved that problem.>I don't believe this is correct -- well, I should rather >say that Quantum theory has nothing to do with it; and >certainly thermodynamics does not view the physical world >as a continuum -- the notion of *molecules* and *atoms*Thermodynamics is mute on that issue. Statistical mechanics is not. And stat mech was closely tied to the notion of atoms, although until 1905 many did not believe in atoms.>certainly has to pre-date Thermodynamics, even if the >classical view of atoms was different than the >post-Quantum Physics view.
Reply by ●October 10, 20052005-10-10
John E. Hadstate wrote:> *** Cross-post to sci.crypt added *** > > "Tim Wescott" <tim@seemywebsite.com> wrote in message > news:Y9WdnYKex68X-9reRVn-pw@web-ster.com... > >>John E. Hadstate wrote: >> >> >>><onyx49@juno.com> wrote in message >>>news:1128730678.728132.286240@g47g2000cwa.googlegroups.com... >>> >>> >>>> >>>>Is there a measure of the "randomness" or should I say >>>> >>>>the "non-randomness" of a signal? >>>> >>>> >>>>Dave >>>> >>> >>> >>>Well, the answer to that is a definite yes, and no. ;) >>> >>>There is a property called "entropy" and it's measured in >>>bits. It's unfortunate that this name collides with the >>>thermodynamic property of the same name, because it >>>confuses a lot of people. >>> >> >>-snip- >> >>It's not an unfortunate name collision, they're the same >>thing (if possibly scaled differently). There's a Richard >>Feynman book on the thermodynamics of computation that >>explains this, and other concepts that seem really twisted >>but really aren't after you think hard enough about them*. > > > They can't be quite the same thing since thermodynamic > entropy measures a property of physical systems while > information theoretic entropy measures a property of > something that is a figment of our imaginations (a useful > figment, to be sure, but "bits" have no physical > manifestation and are not constrained by the laws of > physics). > > I'm not equipped nor motivated to elaborate more precisely > on the difference. That's why I cross-posted the question > to sci.crypt where there are a number of regulars who are > emminently qualified to explain the difference (Doug Gwyn, a > physicist with a good understanding of information theory > comes to mind). >In order to be processed in the physical world bits do, indeed, have to have physical manifestations. At the point where we take pen off of paper and start sending messages (such as this here post) we have left the world of theory and entered the world of physical manifestations -- and at that point our bits become constrained by all the laws of physics that are or have yet to be discovered. -- Tim Wescott Wescott Design Services http://www.wescottdesign.com
Reply by ●October 19, 20052005-10-19
Are there measures of "randomness" or should I say "non-randomness"? Or do people just hold there noses and say "Ewww, this is a pseudo raandom number"? Dave -- comp.lang.c.moderated - moderation address: clcm@plethora.net -- you must have an appropriate newsgroups line in your header for your mail to be seen, or the newsgroup name in square brackets in the subject line. Sorry.
Reply by ●October 20, 20052005-10-20
On Wed, 19 Oct 2005 18:01:07 -0000, onyx49@juno.com wrote:>Are there measures of "randomness" or should I say "non-randomness"?Yes, there are tests based on statistics. If you enter 'randomness measures' into Google, you'll get a number of relevant hits.>Or do people just hold there noses and say >"Ewww, this is a pseudo raandom number"? > >Dave-- Al Balmer Balmer Consulting removebalmerconsultingthis@att.net -- comp.lang.c.moderated - moderation address: clcm@plethora.net -- you must have an appropriate newsgroups line in your header for your mail to be seen, or the newsgroup name in square brackets in the subject line. Sorry.
Reply by ●October 20, 20052005-10-20
onyx49@juno.com wrote:> Are there measures of "randomness" or should I say "non-randomness"? > Or do people just hold there noses and say > "Ewww, this is a pseudo raandom number"? > > Davehttp://www.maa.org/mathland/mathtrek_10_16_00.html might amuse you. For more information, see Knuth; "The Art of Computer Programming", vol. 2 (Seminumerical Algorithms) Section 3.3 (pp. 34-100). Jerry -- Engineering is the art of making what you want from things you can get. ����������������������������������������������������������������������� -- comp.lang.c.moderated - moderation address: clcm@plethora.net -- you must have an appropriate newsgroups line in your header for your mail to be seen, or the newsgroup name in square brackets in the subject line. Sorry.
Reply by ●October 20, 20052005-10-20
onyx49@juno.com wrote:> Are there measures of "randomness" or should I say "non-randomness"? > Or do people just hold there noses and say > "Ewww, this is a pseudo raandom number"? > > DaveThe autocorrelation in this case would be it... If you perform the autocorrelation over a sufficient length of output from the PRNG then you would note that it has several spikes at the cyclic length of the PRNG (ie the length it starts to repeat at). This shows that it's not truly random. A random signal should ideally have a delta at the origin, and 0 everywhere else. It (the autocorrelation function) certainly shouldn't however increase in value at all, it should be monotonically decreasing, if it's not then it suggests some kind of non-randomness. -- comp.lang.c.moderated - moderation address: clcm@plethora.net -- you must have an appropriate newsgroups line in your header for your mail to be seen, or the newsgroup name in square brackets in the subject line. Sorry.
Reply by ●October 25, 20052005-10-25
Bevan Weiss wrote:> onyx49@juno.com wrote: > >> Are there measures of "randomness" or should I say "non-randomness"? >> Or do people just hold there noses and say >> "Ewww, this is a pseudo raandom number"? >> >> Dave > > > The autocorrelation in this case would be it... > If you perform the autocorrelation over a sufficient length of output > from the PRNG then you would note that it has several spikes at the > cyclic length of the PRNG (ie the length it starts to repeat at). This > shows that it's not truly random.Is this the only defect that one usually finds in actual PRNGs?? I'm surprised; I would have expected that the non-top-quality PRNGs would exhibit some noticeable correlation between nearby samples... Am I overestimating the complexity of the problem? Carlos -- -- comp.lang.c.moderated - moderation address: clcm@plethora.net -- you must have an appropriate newsgroups line in your header for your mail to be seen, or the newsgroup name in square brackets in the subject line. Sorry.
Reply by ●October 25, 20052005-10-25
In message <clcm-20051025-0007@plethora.net>, Carlos Moreno <moreno_at_mochima_dot_com@mailinator.com> writes>Bevan Weiss wrote: >> onyx49@juno.com wrote: >> >>> Are there measures of "randomness" or should I say "non-randomness"? >>> Or do people just hold there noses and say >>> "Ewww, this is a pseudo raandom number"? >>> >>> Dave >> The autocorrelation in this case would be it... >> If you perform the autocorrelation over a sufficient length of output >>from the PRNG then you would note that it has several spikes at the >>cyclic length of the PRNG (ie the length it starts to repeat at). >>This shows that it's not truly random. > >Is this the only defect that one usually finds in actual PRNGs?? >I'm surprised; I would have expected that the non-top-quality >PRNGs would exhibit some noticeable correlation between nearby >samples... Am I overestimating the complexity of the problem?There are many tests that show up structure in PRNGs below the level of complete cycles (which might be so long as to never show up) Many of these tests for 'randomness' have been known for a long time and should be applied in the context of the problem where the PRNG is going to be used. You will find details of many tests in 'The Art of Computer Programming' chapter 3 (first chapter of volume 2). While there are positively bad PRNGs there are no universally good ones, only ones that are good in the context of their use. -- Francis Glassborow ACCU Author of 'You Can Do It!' see http://www.spellen.org/youcandoit For project ideas and contributions: http://www.spellen.org/youcandoit/projects -- comp.lang.c.moderated - moderation address: clcm@plethora.net -- you must have an appropriate newsgroups line in your header for your mail to be seen, or the newsgroup name in square brackets in the subject line. Sorry.
