help -- Windzilla (high dynamic range windows)

On Oct 27, 7:00&#4294967295;pm, Fred Marshall <fmarshall_xremove_the...@xacm.org>
wrote:
>
...
> I don't think you can say that:
> i.e.
>
> "Since the DFT of a finite sample set is identical whether the samples
> are from a periodic signal or not"
>
> By definition, the DFT of a finite sample set is "from a periodic
> sequence". &#4294967295;Otherwise, the transform would not be discrete / i.e. sampled.

No, only the continuous/infinite Fourier Transform (FT) must have a
periodicity in one domain to be non-zero only at discrete values in
the other domain.

>
> I know that some disagree with this but I've not absorbed justification
> for that view....
>
> Fred

The conventional DFT is a one to one mapping of N samples in one
domain to N samples in another. There is no information in the
definition of the  of the DFT relating to values of samples at other
values of indices. In this vacuum it has been common to assume the
behavior of the signal reflects the behavior producing the same
results from the FT. Where this is true, it can be a useful
approximation and there are many useful cases where it is true. But
there are many useful cases where the FT assumption is clearly not
true. In dynamic signal analyzers processing samples of real world
signals, the weighted-overlapped spectrum averager (WOSA) method can
be used to reduce the variance of the estimate of the power spectrum.
But WOSA only does this when the signals of interest are stationary
over the period of averaging. When the signal of interest is a long
duration linear FM, we know that the signal is not stationary or
periodic. As I believe Rune has argued in the past, performing a DFT
on a segment of samples forming part of the long sweep does not make
the rest of the samples of the sweep become periodic. That is because
the DFT is not the FT. You have to make your assumptions match the
reality of the data in the discrete world. The N samples in a single
windowed subsegment can't predict the nature of the entire sequence.
This isn't like the FT, it is the real world where we use the DFT on
finite sample sequences.

Dale B. Dalrymple

On Oct 27, 9:03 pm, Fred Marshall <fmarshall_xremove_the...@xacm.org>
wrote:
> On 10/27/2010 7:00 PM, Fred Marshall wrote:
>
>
>
> > On 10/27/2010 4:45 PM, dbd wrote:
>
> >> Since the DFT of a finite sample set is identical whether the samples
> >> are from a periodic signal or not, what would the difference be?
>
> >> Dale B. Dalrymple
>
> > I don't think you can say that:
> > i.e.
>
> > "Since the DFT of a finite sample set is identical whether the samples
> > are from a periodic signal or not"
>
> > By definition, the DFT of a finite sample set is "from a periodic
> > sequence". Otherwise, the transform would not be discrete / i.e. sampled.
>
> > I know that some disagree with this but I've not absorbed justification
> > for that view....
>
> > Fred
>
> What motivates my viewpoint is that the Fourier Transform in whichever
> form is a 1:1 mapping.  That is, given a function you can compute its
> transform and from that transform you can uniquely compute its inverse
> which gets you back to the original function.
>
> So, if when you inverse transform a discrete sequence then you get a
> periodic function and vice versa.

Not with the DFT. With the DFT you can't identify periodicity from a
single DFT'd subsegment.

Consider three sequences:

1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0 ...

1  0 -1  0  1  0 -1  0  1  2  3  4  5  6  7  8  9 10 11 12 ...

1  0 -1  0  1  0 -1  0  0  0  0  0  0  0  0  0  0  0  0  0 ...

All three sequences have the same first 8 samples. Only for the first
sequence does the 8 point DFT of the first 8 samples reflect the
nature of the rest of the signal.

> The issue of dealing with a single period of a periodic function as if
> it's a finite sequence gets folks confused fairly often.
>
> Fred

It is the error of automatically assuming of that the finite sequence
fed to a DFT is a single period of a repeating sequence that gets
folks confused fairly often.

Dale B. Dalrymple

On 10/27/2010 11:01 PM, dbd wrote:
> On Oct 27, 9:03 pm, Fred Marshall<fmarshall_xremove_the...@xacm.org>
> wrote:
>> On 10/27/2010 7:00 PM, Fred Marshall wrote:
>>
>>
>>
>>> On 10/27/2010 4:45 PM, dbd wrote:
>>
>>>> Since the DFT of a finite sample set is identical whether the samples
>>>> are from a periodic signal or not, what would the difference be?
>>
>>>> Dale B. Dalrymple
>>
>>> I don't think you can say that:
>>> i.e.
>>
>>> "Since the DFT of a finite sample set is identical whether the samples
>>> are from a periodic signal or not"
>>
>>> By definition, the DFT of a finite sample set is "from a periodic
>>> sequence". Otherwise, the transform would not be discrete / i.e. sampled.
>>
>>> I know that some disagree with this but I've not absorbed justification
>>> for that view....
>>
>>> Fred
>>
>> What motivates my viewpoint is that the Fourier Transform in whichever
>> form is a 1:1 mapping.  That is, given a function you can compute its
>> transform and from that transform you can uniquely compute its inverse
>> which gets you back to the original function.
>>
>> So, if when you inverse transform a discrete sequence then you get a
>> periodic function and vice versa.
>
> Not with the DFT. With the DFT you can't identify periodicity from a
> single DFT'd subsegment.
>
> Consider three sequences:
>
> 1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0 ...
>
> 1  0 -1  0  1  0 -1  0  1  2  3  4  5  6  7  8  9 10 11 12 ...
>
> 1  0 -1  0  1  0 -1  0  0  0  0  0  0  0  0  0  0  0  0  0 ...
>
> All three sequences have the same first 8 samples. Only for the first
> sequence does the 8 point DFT of the first 8 samples reflect the
> nature of the rest of the signal.
>
>> The issue of dealing with a single period of a periodic function as if
>> it's a finite sequence gets folks confused fairly often.
>>
>> Fred
>
> It is the error of automatically assuming of that the finite sequence
> fed to a DFT is a single period of a repeating sequence that gets
> folks confused fairly often.
>
> Dale B. Dalrymple

Dale,

Thanks for the ideas:

Your perspective is useful and different from mine.

'When you say:
"Only for the first sequence does the 8 point DFT of the first 8 samples 
reflect the nature of the rest of the signal."

And, with the " ... ' at the ends it clearly shows you have 3 different 
"signals" that have extent beyond the first 8 samples.  And, there are 
likely lots of things that you can say about each of these signals.

I would assert that when you decide to take an 8-sample DFT then you are 
making an important decision about what that does.  In effect, you are 
throwing away all that other signal information and have no idea if the 
8 samples you've taken will repeat or be something entirely different 
*had you* taken more samples.
AND, on top of that, one imposes a constraint that the samples taken are 
one period of an endless periodic sequence.

You say:
"It is the error of automatically assuming of that the finite sequence
fed to a DFT is a single period of a repeating sequence that gets
folks confused fairly often."

so, from above, I obviously can't agree with that.  Here is my 
arm-waving "proof":

I am going to assume that the Fourier Transform or Discrete Fourier 
Transform, either one is a 1:1 mapping.  To make any other assumption I 
can only imagine would be disastrous and I really don't want to 
contemplate that.  If the mapping isn't 1:1 then all sorts of nonlinear 
things happen I think.  I'm sorry I'm not more articulate on this matter.

So, when I take 8 samples and perform a DFT, I get another 8 (complex) 
samples as output.

Now, I may perform an Inverse *Fourier Transform* on those 8 complex 
spectral samples and will get a continuous, periodic waveform.  Same as 
constructing from a Fourier Series set of coefficients.  I think this 
observation has bearing.

Or, I may perform an Inverse *DFT* on those 8 complex spectral samples 
and will get a single period of a discrete periodic sequence.

If one samples the result of the IFT one will get the result of the IDFT 
.. I think.

**

The difference seems to be that you say it's an error: "assuming that 
the finite sequence fed to a DFT is a single period ...."
and I would say
"because the finite sequence fed to a DFT *is* a single period by 
definition..." where no assumption is involved.

Often we get ourselves messed up here because one talks about some 
practical thing and another talks about an analytical thing or sometimes 
one of us talks about both in the same paragraph without being clear 
which one is talking about.

Here, your example seems to be in a practical context.  It can really 
happen that way.
And, my context is the analytical one and I totally disregard what 
remains of those 3 signals because they "don't exist" any more for me. 
Surely one can't "go back" to them, eh?

If I didn't say it before, I really liked your paper!  Great treatment.


Fred

On 10/27/2010 10:38 PM, dbd wrote:
> No, only the continuous/infinite Fourier Transform (FT) must have a
> periodicity in one domain to be non-zero only at discrete values in
> the other domain.

This confuses me.  Does this mean that the DFT need not have a 
periodicity in one domain to be non-zero only at discrete values in the 
other domain or what exactly?

I'm not being dense, I just don't get it yet.

Fred

On 10/27/2010 10:38 PM, dbd wrote:


Predicting is a practical endeavor.

I was only computing transforms.

Fred

On Oct 28, 3:46&#4294967295;pm, Fred Marshall <fmarshall_xremove_the...@xacm.org>
wrote:
...
> Dale,
>
> Thanks for the ideas:
>
> Your perspective is useful and different from mine.
>
> 'When you say:
> "Only for the first sequence does the 8 point DFT of the first 8 samples
> reflect the nature of the rest of the signal."
>
> And, with the " ... ' at the ends it clearly shows you have 3 different
> "signals" that have extent beyond the first 8 samples. &#4294967295;And, there are
> likely lots of things that you can say about each of these signals.

Since the "..." comes after 20 samples, not just 8 we definitely can
say that the 2nd and 3rd sequences are not periodic in a period of
less than 20 samples. Yet, as I stated, the DFT of the first 8 samples
of each of the sequences is identical.

>
> I would assert that when you decide to take an 8-sample DFT then you are
> making an important decision about what that does. &#4294967295;In effect, you are
> throwing away all that other signal information and have no idea if the
> 8 samples you've taken will repeat or be something entirely different
> *had you* taken more samples.
> AND, on top of that, one imposes a constraint that the samples taken are
> one period of an endless periodic sequence.

There is an obvious bizarre illogic to claiming that after windowind
to 8 samples we have no knowledge. (I have agreed with that.) and then
insisting, without knowledge, that the 8 samples represent a periodic
signal. Two out of my three examples show that that isn't true.

The "one imposes a constraint ..." is a constraint that has no basis
in the samples.
Sometimes it is reasonable and sometimes it nonsense.

> ...

>
> Or, I may perform an Inverse *DFT* on those 8 complex spectral samples
> and will get a single period of a discrete periodic sequence.

There is no rational basis for an assumption of periodicity. It may be
true, it may not and it depends on outside information.

>
> If one samples the result of the IFT one will get the result of the IDFT
> .. I think.
>
> **
>
> The difference seems to be that you say it's an error: "assuming that
> the finite sequence fed to a DFT is a single period ...."
> and I would say
> "because the finite sequence fed to a DFT *is* a single period by
> definition..." where no assumption is involved.

This "single period by definition" is not valid in 2 of my 3 examples.

>
> Often we get ourselves messed up here because one talks about some
> practical thing and another talks about an analytical thing or sometimes
> one of us talks about both in the same paragraph without being clear
> which one is talking about.
>
> Here, your example seems to be in a practical context. &#4294967295;It can really
> happen that way.
> And, my context is the analytical one and I totally disregard what
> remains of those 3 signals because they "don't exist" any more for me.

Then you conclusions are  irrational, the other samples do exist in
the real world.

> Surely one can't "go back" to them, eh?

Surely one can and does. In a dynamic signal analyzer, I can slide the
window half the transform size and calculate another DFT (50%
overlap). This continues with each DFT calculating a new line in a
waterfall or spectrogram display. I have the rest of the samples and I
can determine whether the "periodic assumption" is valid or not. In
real world data sets it almost never is even close. There is
information, the other samples, outside the window that allows
decisions about periodicity to be evaluated.

That's the real world. It doesn't go away and not come back each time
you apply a window to a sequence. Continuing to use the information
outside each window application is what signal processing systems
actually do.

>
> If I didn't say it before, I really liked your paper! &#4294967295;Great treatment.
>
> Fred

Thank you.

Dale B. Dalrymple

On Oct 28, 4:07&#4294967295;pm, Fred Marshall <fmarshall_xremove_the...@xacm.org>
wrote:
> On 10/27/2010 10:38 PM, dbd wrote:
>
> > No, only the continuous/infinite Fourier Transform (FT) must have a
> > periodicity in one domain to be non-zero only at discrete values in
> > the other domain.
>
> This confuses me. &#4294967295;Does this mean that the DFT need not have a
> periodicity in one domain to be non-zero only at discrete values in the
> other domain or what exactly?
>
> I'm not being dense, I just don't get it yet.
>
> Fred

If the FT has non-zero values at the sample times and zero values in
between sample times the continuous time signal must have been
periodic. Infinitely extending periodicity is what is required for the
intervals between sample times to be zero. In the DFT there are no
values between sample times.  There is neither need nor justification
for a periodic constraint.

Dale B. Dalrymple

On Oct 28, 4:08&#4294967295;pm, Fred Marshall <fmarshall_xremove_the...@xacm.org>
wrote:
> On 10/27/2010 10:38 PM, dbd wrote:
>
> Predicting is a practical endeavor.
>
> I was only computing transforms.
>
> Fred

I'm sorry Fred. I can't follow this. It has three lines but it doesn't
seem to scan correctly for a haiku.

Dale B. Dalrymple

Fred Marshall  <fmarshall_xremove_the_xs@xacm.org> wrote:

>On 10/27/2010 10:38 PM, dbd wrote:

>> No, only the continuous/infinite Fourier Transform (FT) must have a
>> periodicity in one domain to be non-zero only at discrete values in
>> the other domain.

>This confuses me.  Does this mean that the DFT need not have a 
>periodicity in one domain to be non-zero only at discrete values in the 
>other domain or what exactly?

>I'm not being dense, I just don't get it yet.

I would say that a signal which is time-limited to an interval has
the same information content as an otherwise identical signal that
is periodic with period equal to the interval... so whether a transform
requires one or the other as its input is entirely a matter of 
convention / terminology / notation.

(Not that these things are unimportant.  They are just not information.)


Steve

On 10/28/2010 6:45 PM, dbd wrote:
Dale,

Well, it's clear that we're each stuck on a particular perspective.
I don't think there's anything wrong with your perspective but I believe 
it leads to some unfortunate terminology.

When you say "a period", it appears you're referring to actual periodic 
content in a signal of some length.  That's very understandable and fine.

When I've been saying "a period" in the context of this thread, I'm 
referring to the implicit period determined by the number of samples 
used in the DFT.
And, I don't think it's nonsense to say that once that's set there is no 
going back *with that particular sequence of data that has been grabbed 
(not where it was grabbed from)*.  That's what I meant.

I do have a little trouble with your treatment of the sliding DFTs in an 
analyzer context:

Let's suppose we have streaming input data samples.
Let's suppose the samples are at fs=1.024kHz with sampling interval 
.999023438 msec.
Let's suppose we will grab 1,024 samples for a DFT.
So the temporal span of the grab is 0.999023438 seconds and represents 
1.000... seconds in a DFT/IDFT temporal period context.

So, the period in time is 1.0000.. seconds and the period in frequency 
is 1,024 Hz and the frequency sample interval is 1 Hz.

Let's now assume that there's a signal of interest at 400Hz and we 
perform DFTs at 0.5 second intervals on 1024 samples.  The period of 
400Hz is 2.5msec and the frequency index of the 400Hz sample will be 400 
if we start at 0.

I don't see how there being a 1Hz temporal period implicit in the DFT 
nor a 1024 Hz frequency period implicit in the DFT have any impact or 
bearing on this..... either the 400Hz is there or it isn't and, yes, it 
may fade and one can generate waterfalls, etc. etc.  but that really has 
nothing to do with the 1024Hz period here.

But, if the signal component of interest had a period that's longer than 
1 second then there are all kinds of things going on that go beyond this 
discussion because of aliasing, etc.  Your examples were sorta like 
this.  If the *underlying signal period* is longer than the sample 
sequence interval then there will be aliasing and under-resolution.

And, finally, if one takes a fairly arbitrary window, one knows that 
there can be actual spectral spreading as well as theoretically possible 
spectral spreading ... which is really another word for aliasing.

By saying "you can't go back" I didn't mean "you can't go forward" and 
get more information....  I meant you're still constrained by the time 
window and the frequency window.  And the samples are the samples. 
That's always the case.

For example, if I had 400.5 Hz above then I wouldn't be able to resolve 
to 400.5 Hz with the parameters given without computing an estimate.  If 
noiseless then I might be able to do a reasonable estimate with a single 
frequency sequence.  If noisy then I might want to have more frequency 
sequences to work on.  But, the 1Hz limit looms large in this endeavor. 
  .... and the 1,024 Hz period is a fundamental limit in it all.

Fred