Show me some more numbers

Cedron <103185@DSPRelated> wrote:

>>Then replace "error rate" by "average error" (i.e your first column
>>of data).

>>> So it is just a matter of tightening up the variance.  There
>>> are just as many expected cases that need to be increased as
>>> those that need to be decreased.  So by "cooking" the numbers
>>> you can expect values that will be more similar to making a
>>> larger set of runs.

>>I don't think so.

>By variance here I meant the variance of the average error values.  In
>your own words, cooking the noise will make the average error values more
>optimistic, meaning closer to zero.  This is what a larger set of runs is
>expected to do as well.

Just because two series both trend towards zero does not mean
they have the same statistics.

In this case, your series has the wrong statistics.

>>By rescaling each individual noise pattern to have the same sigma, you
>>are destroying your results.

>"Modifying" is not "destroying".

I'm sticking with "destroying" in this instance.

In many situations with AWGN noise, it is a fraction of outlying 
cases with large variance that dominates performance.  If you 
specifically rescale these cases downward, you are destroying
your data.  (I tried to get at this point earlier, with my
examples where 10 or 12 noise patterns out of 10,000 might
dominate performance.  This is a very real-world effect in
addition to being an expected theoretircal effect.)

>>The purpose of creating an ensemble of 1,000 (or 10,000, or as 
>>many as is necessary) noise patterns is so that you can see how the 
>>statistics of added white Gaussian noise affect system performance.  

>Unfortunately, the precision issue messes with this.

If true you then need to work on your simulator implementation.

>Well, I never claimed AWGN, all I claimed was "near Gaussian".  

That's okay to claim this, but it does not prove that any
"near-Gaussian" method is valid.

It's not so difficult to do it right.  It's way less of an intellectual
challenge than you faced when deriving your formulae in the first
place, so I cannot understand your resistance on this one.

Steve

>Cedron <103185@DSPRelated> wrote:
>
>>>Then replace "error rate" by "average error" (i.e your first column
>>>of data).
>
>>>> So it is just a matter of tightening up the variance.  There
>>>> are just as many expected cases that need to be increased as
>>>> those that need to be decreased.  So by "cooking" the numbers
>>>> you can expect values that will be more similar to making a
>>>> larger set of runs.
>
>>>I don't think so.
>
>>By variance here I meant the variance of the average error values.  In
>>your own words, cooking the noise will make the average error values
more
>>optimistic, meaning closer to zero.  This is what a larger set of runs
is
>>expected to do as well.
>
>Just because two series both trend towards zero does not mean
>they have the same statistics.
>

"More similar" is not a claim of "sameness".  

>In this case, your series has the wrong statistics.
>

I find your assertion of "wrong" disturbing.  The strongest words you
should use are "distorted compared to AWGN."  Even so, adding a "somewhat"
in front would be appropriate.

>>>By rescaling each individual noise pattern to have the same sigma, you
>>>are destroying your results.
>
>>"Modifying" is not "destroying".
>
>I'm sticking with "destroying" in this instance.
>
>In many situations with AWGN noise, it is a fraction of outlying 
>cases with large variance that dominates performance.  If you 
>specifically rescale these cases downward, you are destroying
>your data.  (I tried to get at this point earlier, with my
>examples where 10 or 12 noise patterns out of 10,000 might
>dominate performance.  This is a very real-world effect in
>addition to being an expected theoretircal effect.)
>

You can expect extreme cases to be tempered somewhat, but not as much as
you imply.  If we were looking for outliers your point might be relevant,
but since the only thing being measured is average and standard deviation
(of the transformed noise), the effect will be minimal.

>>>The purpose of creating an ensemble of 1,000 (or 10,000, or as 
>>>many as is necessary) noise patterns is so that you can see how the 
>>>statistics of added white Gaussian noise affect system performance.  
>

AWGN is a mathematical model with nice statistical properties.  The main
difference between the near Gaussian model I used and a true Gaussian
distribution is that the near one is bounded.  To me, this is more likely
to be representative of real life noise.  I would expect that near
Gaussian is closer to AWGN than real life noise is.  I would also expect
that real life noise is more similar to near Gaussian than AWGN.

>>Unfortunately, the precision issue messes with this.
>
>If true you then need to work on your simulator implementation.
>

This is a warning to anyone who thinks that double precision floating
point will be correct for large numbers of runs of any test type.  If it
hadn't been for the fact that my three bin complex formula is equivalent
to Candan's 2013, I would not have seen this or suspected it.

>>Well, I never claimed AWGN, all I claimed was "near Gaussian".  
>
>That's okay to claim this, but it does not prove that any
>"near-Gaussian" method is valid.
>

Your conflating of "correct, valid, right" with "conventional, standard"
is really bothersome to me.  It's like you're an uber-conformist, and you
insist everyone else should be too.  If you asked me what the square root
of nine was and I said negative three, my answer would be "correct, valid,
and right", but not "conventional nor standard".

>It's not so difficult to do it right.  It's way less of an intellectual
>challenge than you faced when deriving your formulae in the first
>place, so I cannot understand your resistance on this one.
>
>Steve

It's not a matter of resistance.  I came up with a test, explained it, and
showed the results.  What is hard to understand is why you insist that it
is not good enough and must be done to a certain convention or it is
unacceptable.  The conclusions that can be drawn from either scenario are
not going to differ.  I shouldn't have had to do any testing at all.  The
reason I did it was to counter Jacobsen's arrogance in the Matlab Beginner
thread when he said he had seen formulas that claim to be exact come and
go and fall apart when presented with noise and thus could not be bothered
to examine it.  Even after I told him that his estimator was an
approximation of my formula and thus similar behavior when exposed to
noise should be expected.

Ced
---------------------------------------
Posted through http://www.DSPRelated.com

Hi, Cedron

  It seems to me that a similar frequency estimation problem was
considered  by so called the high resolution spectrum estiamtion,
especially for the case of a single frequency signal. You could google
Pisarenko, MUSIC or the eigen vector based spectrum estimation. The
Pisaredko frequency estimation is given by

  w=acos((r2+sqrt(r2^2+8r1^2))/(4r1));

where r1 and r2 are the autocorrelation function of the signal. 

  It is interested in to see how your approach against the Pisarenko
approach.

Kathy

---------------------------------------
Posted through http://www.DSPRelated.com

Cedron <103185@DSPRelated> wrote:

> Pope wrote,

>>In this case, your series has the wrong statistics.

>I find your assertion of "wrong" disturbing.  The strongest words you
>should use are "distorted compared to AWGN."  Even so, adding a "somewhat"
>in front would be appropriate.

>>>>By rescaling each individual noise pattern to have the same sigma, you
>>>>are destroying your results.

>>>"Modifying" is not "destroying".

>>I'm sticking with "destroying" in this instance.

>>In many situations with AWGN noise, it is a fraction of outlying 
>>cases with large variance that dominates performance.  If you 
>>specifically rescale these cases downward, you are destroying
>>your data.  (I tried to get at this point earlier, with my
>>examples where 10 or 12 noise patterns out of 10,000 might
>>dominate performance.  This is a very real-world effect in
>>addition to being an expected theoretircal effect.)

>You can expect extreme cases to be tempered somewhat, but not as much as
>you imply.  If we were looking for outliers your point might be relevant,
>but since the only thing being measured is average and standard deviation
>(of the transformed noise), the effect will be minimal.

(Tangentially, "average" (mean) frequency error is a valid reduced 
statistic, but many would be more interested in the RMS frequency error, 
which is going to be more influenced by infrequent yet large error values.)

But more important, to me anyway, is simulating AWGN noise to a
reasonble precision; other types of noise are valid, but in almost
every instance AWGN is the single most interesting case, because
AWGN is "worst case" noise -- it impairs the achievable results more than 
any other type of noise of the same noise power. It also arises
in nature by any of a large number of mechanisms (e.g. receiver
equivalent input noise).  So it's the first case most investigators
will look at, and it is usually the case you characterize before
looking at more exotic forms of noise.

>AWGN is a mathematical model with nice statistical properties.  The main
>difference between the near Gaussian model I used and a true Gaussian
>distribution is that the near one is bounded.  

That is okay, perhaps depending on details (all simulated AWGN generators
will be bounded by something).

It's really the per-pattern noise-rescaling idea that bothers
me, since it likely significantly affects the impairments, but I 
think you've said you haven't actually done this in your sims, it's 
just something you propose as improving the interpretation of 
certain results ... I disagree strongly, but no big deal.

>>>Unfortunately, the precision issue messes with this.

>>If true you then need to work on your simulator implementation.

>Your conflating of "correct, valid, right" with "conventional, standard"
>is really bothersome to me.  It's like you're an uber-conformist, and you
>insist everyone else should be too.  

I believe conventions are very useful for creating a baseline from
which investigations can proceed.  The convention of characterizing
first against AWGN, before jumping into variants, is supported by a 
lot of theory, a lot of practical considerations, *and* in
addition to that a lot of convention.  It's not just an isolated,
randomly-chosen convention.  I think/hope if you appreciated
this you would not see me as a "uber-conformist" but more as
a comm engineer with normal sensibilities about how best
to process with simulation experiments.

Steve

>Hi, Cedron
>
>  It seems to me that a similar frequency estimation problem was
>considered  by so called the high resolution spectrum estiamtion,
>especially for the case of a single frequency signal. You could google
>Pisarenko, MUSIC or the eigen vector based spectrum estimation. The
>Pisaredko frequency estimation is given by
>
>  w¬os((r2+sqrt(r2^2+8r1^2))/(4r1));
>
>where r1 and r2 are the autocorrelation function of the signal. 
>
>  It is interested in to see how your approach against the Pisarenko
>approach.
>
>Kathy
>
>---------------------------------------
>Posted through http://www.DSPRelated.com

Hi Kathy,

I am not familiar with the Pisarenko method so I can't answer your
question myself.  What I did notice was that the equation you gave is
clearly the root of a quadratic equation.  So I did a little researching.

First, I read the Wikepedia and found that is was for complex signals.

Then I found this paper from 2002:

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.78.5602&rep=rep1&type=pdf

The thing about this paper was that it tested against real valued
signals.

Then I found this paper from 2007 with one of the same authors:

http://www.ee.cityu.edu.hk/~hcso/eusipco07_3.pdf

"Cross-multiplying (3) and (4), we obtain the following approximate
equation:

2r1 cos^2(w) - r2 cos(w) - r1 = 0       (5)"

There is the quadratic equation, but notice the keyword "approximate". 
Again, they are studying a real signal.  They also come up with an
improved formula that according to their figures performs much better.

I have come up with 3 bin DFT formulas for both the real and complex case.
 They are both exact in the noiseless case and do quite well in the
presence of noise.  A very nice paper produced by Julien Arzi with
comparisons of my formulas and a slew of others against both real and
complex signals can be found at this site:

http://www.tsdconseil.fr/log/scriptscilab/festim/index-en.html

Follow the link on the right side labelled:

"Comparison of different frequency estimation algorithms (pdf)"

The charts in Julien's analysis are similar to the ones in the papers I
cited above except the vertical scales are given in powers of ten rather
than dbs.

The derivation of my real valued signal formula can be found at:

http://www.dsprelated.com/showarticle/773.php

I have not published the derivations of my 2 bin or 3 bin complex valued
signal formulas.  I have not revealed my 2 bin real valued signal formula
yet.  I will do so when Martin Vicanek finishes his work.

His paper can be found at:

http://vicanek.de/dsp/FreqFromTwoBins.pdf

Don't pay any serious attention to my results posted in this or the "Show
me the numbers" thread.  They are meant to be indicative, they are not
comprehensive, i.e. only one frequency range with a fixed phase value, and
as you can tell by my discussion with Steve Pope, the noise model is quite
suspect.

Thanks for bringing up the Pisarenko method.  Maybe Julien will add it to
his analysis and you can get a direct answer to your question.

Ced

---------------------------------------
Posted through http://www.DSPRelated.com

[...snip...]

>
>(Tangentially, "average" (mean) frequency error is a valid reduced 
>statistic, but many would be more interested in the RMS frequency error,

>which is going to be more influenced by infrequent yet large error
values.)
>

Well, since the (arithmetic) average (aka mean) is so small compared to
the standard deviation in these runs, its variance from zero is hardly
significant and the standard deviation is a very good approximation of the
RMS.  I reported the standard deviation rather than the RMS because of its
independence of the mean.


>But more important, to me anyway, is simulating AWGN noise to a
>reasonble precision; other types of noise are valid, but in almost
>every instance AWGN is the single most interesting case, because
>AWGN is "worst case" noise -- it impairs the achievable results more than

>any other type of noise of the same noise power. It also arises
>in nature by any of a large number of mechanisms (e.g. receiver
>equivalent input noise).  So it's the first case most investigators
>will look at, and it is usually the case you characterize before
>looking at more exotic forms of noise.
>
I think my near Gaussian approach is actually a pretty close approximation
of AWGN.

>>AWGN is a mathematical model with nice statistical properties.  The
main
>>difference between the near Gaussian model I used and a true Gaussian
>>distribution is that the near one is bounded.  
>
>That is okay, perhaps depending on details (all simulated AWGN
generators
>will be bounded by something).
>

Well, at least by the precision of the variable holding it.  ;-)

>It's really the per-pattern noise-rescaling idea that bothers
>me, since it likely significantly affects the impairments, but I 
>think you've said you haven't actually done this in your sims, it's 
>just something you propose as improving the interpretation of 
>certain results ... I disagree strongly, but no big deal.
>

I did try it.  Didn't change much.

>
>I believe conventions are very useful for creating a baseline from
>which investigations can proceed.  The convention of characterizing
>first against AWGN, before jumping into variants, is supported by a 
>lot of theory, a lot of practical considerations, *and* in
>addition to that a lot of convention.  It's not just an isolated,
>randomly-chosen convention.  I think/hope if you appreciated
>this you would not see me as a "uber-conformist" but more as
>a comm engineer with normal sensibilities about how best
>to process with simulation experiments.
>
>Steve

I am the veteran of many "Standards vs Diversity" discussions in many
different contexts.  You cross a bright line when you say something that
is non-standard is wrong.  Conventions are rarely baseless, but if you
don't understand the basis of a convention and follow it blindly, well
then, that's as bad as blindly following policy without understanding the
rationale behind it.

I don't have to sell you on the benefits of standardization.  How about
the down side?  Standards are a security risk.  Standards are an
impediment to innovation.

I'm a mathematician who leans towards the side of non-conformist.  More
room for creativity there.

Ced

---------------------------------------
Posted through http://www.DSPRelated.com

Cedron <103185@DSPRelated> wrote:

> Pope wrote,

>>But more important, to me anyway, is simulating AWGN noise to a
>>reasonble precision; other types of noise are valid, but in almost
>>every instance AWGN is the single most interesting case, because
>>AWGN is "worst case" noise -- it impairs the achievable results more than
>>any other type of noise of the same noise power. It also arises
>>in nature by any of a large number of mechanisms (e.g. receiver
>>equivalent input noise).  So it's the first case most investigators
>>will look at, and it is usually the case you characterize before
>>looking at more exotic forms of noise.

>I think my near Gaussian approach is actually a pretty close approximation
>of AWGN.

>>It's really the per-pattern noise-rescaling idea that bothers
>>me, since it likely significantly affects the impairments, but I 
>>think you've said you haven't actually done this in your sims, it's 
>>just something you propose as improving the interpretation of 
>>certain results ... I disagree strongly, but no big deal.

>I did try it.  Didn't change much.

Okay, but this could be another artifact of too-short runsizes.

>>I believe conventions are very useful for creating a baseline from
>>which investigations can proceed.  The convention of characterizing
>>first against AWGN, before jumping into variants, is supported by a 
>>lot of theory, a lot of practical considerations, *and* in
>>addition to that a lot of convention.  It's not just an isolated,
>>randomly-chosen convention.  I think/hope if you appreciated
>>this you would not see me as a "uber-conformist" but more as
>>a comm engineer with normal sensibilities about how best
>>to process with simulation experiments.

>I am the veteran of many "Standards vs Diversity" discussions in many
>different contexts.  You cross a bright line when you say something that
>is non-standard is wrong.  Conventions are rarely baseless, but if you

(Standards != conventions)

>don't understand the basis of a convention and follow it blindly, well
>then, that's as bad as blindly following policy without understanding the
>rationale behind it.

>I don't have to sell you on the benefits of standardization.  How about
>the down side?  Standards are a security risk.  Standards are an
>impediment to innovation.

>I'm a mathematician who leans towards the side of non-conformist.  More
>room for creativity there.

This seems like rather a lot of dancing around rather than admitting
to even a slight possibility that you are not promoting the
most sound experimental approach.

You have done a great job in deriving your formulae.  You did
a pretty reasonable job demonstrating their performance.  It's
your philosophy of science of which I am unconvinced, 

Steve

[...snip...]
>
>>I am the veteran of many "Standards vs Diversity" discussions in many
>>different contexts.  You cross a bright line when you say something
that
>>is non-standard is wrong.  Conventions are rarely baseless, but if you
>
>(Standards != conventions)
>

This statement got me thinking.  I should have been saying "common
conventions" when I used the word conventions.  Standards are codified
conventions, thus a subset of common conventions.

So, you also cross a bright line when you say something that is
unconventional is wrong just because it is unconventional.

[...snip...]
>
>This seems like rather a lot of dancing around rather than admitting
>to even a slight possibility that you are not promoting the
>most sound experimental approach.
>

Calling it "adequate" is hardly claiming it is the most sound simulation
approach.  I have not been promoting it, I have been defending it.  There
is more than a slight possibility that it is not the soundest approach. 
However, I don't think a more sound approach would draw any different
conclusions.

>You have done a great job in deriving your formulae.  You did
>a pretty reasonable job demonstrating their performance.  It's
>your philosophy of science of which I am unconvinced, 
>
>Steve

Thanks for the compliment.  I am still waiting for somebody to step up and
say they are familiar with "the gospel" and my formulas aren't in there. 
I'll accept "pretty reasonable" as a synonym for "adequate", so no
disagreement there.  I'll also point out that this isn't Science, it's
applied mathematics.

Ced
---------------------------------------
Posted through http://www.DSPRelated.com

Cedron <103185@DSPRelated> replies to my post,

>>(Standards != conventions)

> This statement got me thinking.  I should have been saying "common
>conventions" when I used the word conventions.  Standards are codified
>conventions, thus a subset of common conventions.

Okay

>So, you also cross a bright line when you say something that is
>unconventional is wrong just because it is unconventional.

I did not say that downplaying the importance of applying AWGN
to a system sim is wrong because it is unconventional, it
is wrong because it is undershooting the technical goals of doing
the simulation in the first place.

It is a convention, yes, but it is a convention because the
science of the situation indicates that it is important.
You seem to be recasting this as me blindly following a convention;
that is basically a content-free argument on your part that could be 
used against any situation where somebody is following a consensus
point of view, whether what they are doing makes scientific sense 
or not.


So, yes, while it's important to battle consensus science sometimes,
this is not one of those times.

An analogy: let's suppose you need to measure the distance between
two points on a plane, and your only tool is a yardstick.  I
suggest you iterate by laying the yardstick end-to-end and counting
until you have measured the distance in yards.  Let's say the answer is 50 yards.

You could say I have "crossed a bright line" by suggesting you
must count all 50 repetitions and not just, say, 42.   Or that
some of the "yards " should only count as three-quarters of a 
yard.

This would be analogous to saying that some of the AWGN datapoints
in a sim could be left out, or re-scaled.   Is it valid to say
counting all 50 yards is consensus science and therefore not to
be trusted?

Hey, good discussion.



Steve

[...snip...]
>
>I did not say that downplaying the importance of applying AWGN
>to a system sim is wrong because it is unconventional, it
>is wrong because it is undershooting the technical goals of doing
>the simulation in the first place.
>

The purpose of these simulations was to do a side by side comparison of
the formulas in the presence of noise.  There were no other technical
goals, as in comparing the results to any other evaluation.  For this
goal, knowing the noise type is less important since none of the
comparison criteria depends on a solid knowledge of the noise.

>It is a convention, yes, but it is a convention because the
>science of the situation indicates that it is important.
>You seem to be recasting this as me blindly following a convention;
>that is basically a content-free argument on your part that could be 
>used against any situation where somebody is following a consensus
>point of view, whether what they are doing makes scientific sense 
>or not.
>

The screech of the uber-conformist: "That's how everybody does it."  Just
kidding.  I am not saying, nor have I, that following a convention is
wrong.  What I am saying is judging an approach that doesn't follow
convention as wrong just because it doesn't follow convention is wrong.  I
am also saying, just as with policy, is you should understand the
rationale for the convention to know why it is good to follow.  In your
case, in our discussion, you have endeavoured to do that, I am not trying
to cast you as blindly following common convention.  What I am faulting
you for is your harsh terminology for techniques that don't follow
convention.

>
>So, yes, while it's important to battle consensus science sometimes,
>this is not one of those times.
>

Anybody want to talk about Anthropogenic Global Warming?

>An analogy: let's suppose you need to measure the distance between
>two points on a plane, and your only tool is a yardstick.  I
>suggest you iterate by laying the yardstick end-to-end and counting
>until you have measured the distance in yards.  Let's say the answer is
50
>yards.
>
>You could say I have "crossed a bright line" by suggesting you
>must count all 50 repetitions and not just, say, 42.   Or that
>some of the "yards " should only count as three-quarters of a 
>yard.
>

False analogy.  Suppose I used the ruler and laid out a 3-4-5 triangle. 
Then I completed the rectangle so one corner was on one point to be
measured and the long side of the rectangle pointed right to the other
away point.  I then put my ruler on corner perpendicular to the first
point and aim my ruler at the away point.  I mark the crossing point on
the other side of the rectangle, then use similar triangles to calculate
the distance to the away point.

Foul, you scream, that is not what I suggested.  It is wrong!

>This would be analogous to saying that some of the AWGN datapoints
>in a sim could be left out, or re-scaled.   Is it valid to say
>counting all 50 yards is consensus science and therefore not to
>be trusted?

Now, you are once again inferring that I said the consensus is not to be
trusted.

On the rescaling issue, what I said, and I stand by it, though I haven't
actually proved it, is that the numerical values of a smaller run with the
normalization rescaling will be closer to the values of unnormalized
results of a longer run than a smaller run with unnormalized values.  If
you want to numerically contest that, it's your turn to do some coding.

>
>Hey, good discussion.
>

Yep, I hope others are enjoying it too.

>
>
>Steve

Ced
---------------------------------------
Posted through http://www.DSPRelated.com