Purpose of Detrending signal before FFT?

dbd <dbd@ieee.org> wrote:
> On Tuesday, October 2, 2012 9:13:07 AM UTC-7, Tim Wescott wrote:
> ...
 
>> If you're windowing you need to remove the mean, or you just end up with 
>> the DFT of the window.

> It's silly posts like this that make Vlad post the way he does.

I had to think about this one for at least a few seconds.

For short sequences and windows, it is pretty true.

For longer ones, less true. Well, a window might round of just the
end points to a long sequence of data points, which won't have
a big effect on other than the 0 componens, but a really big
effect on a short sequence.

-- glen

On Thursday, October 4, 2012 1:13:28 PM UTC-7, Tim Wescott wrote:
...

> So, having established your expertise by insult, please back up that 
> statement!

And what backup did you give your statement?

> 
> Show how taking the DFT of a sample with a small signal embedded in a 
> honkin' big DC offset, windowed, _doesn't_ have a huge contribution from 
> the window!
> 

Your simple statement didn't include this qualification or the small DFT size case glen discusses. (Or the limited resolution situation case which becomes important if you include the big signal qualification.)

The DFT is a linear operator. The mean of the time domain data is the DC term in the frequency domain. Subtraction of this component from both domains does not alter any term in the frequency domain but the DC term.

> ... particularly in that it is something that's 
> being pointed out to an obviously rank beginner who clearly hasn't had 
> much mileage at all with using the DFT for signal analysis.

You post a lot here and often usefully. If a stupid poster had posted something stupid I might not comment. You know how to post coherently. You didn't here. Exactly because of the OP's inexperience, I felt it worth a quick kick.

> Tim Wescott

Dale B. Dalrymple

On Fri, 05 Oct 2012 14:37:25 -0700, dbd wrote:

> On Thursday, October 4, 2012 1:13:28 PM UTC-7, Tim Wescott wrote: ...
> 
>> So, having established your expertise by insult, please back up that
>> statement!
> 
> And what backup did you give your statement?
> 
> 
>> Show how taking the DFT of a sample with a small signal embedded in a
>> honkin' big DC offset, windowed, _doesn't_ have a huge contribution
>> from the window!
>> 
>> 
> Your simple statement didn't include this qualification or the small DFT
> size case glen discusses. (Or the limited resolution situation case
> which becomes important if you include the big signal qualification.)
> 
> The DFT is a linear operator. The mean of the time domain data is the DC
> term in the frequency domain. Subtraction of this component from both
> domains does not alter any term in the frequency domain but the DC term.
> 
>> ... particularly in that it is something that's being pointed out to an
>> obviously rank beginner who clearly hasn't had much mileage at all with
>> using the DFT for signal analysis.
> 
> You post a lot here and often usefully. If a stupid poster had posted
> something stupid I might not comment. You know how to post coherently.
> You didn't here. Exactly because of the OP's inexperience, I felt it
> worth a quick kick.
> 
>> Tim Wescott
> 
> Dale B. Dalrymple

Well, you seem to be missing what I was talking about.  If you apply a 
_window_ to data with a DC bias, then your result after DFT is going to 
be the DFT of the window, plus the DFT of the "interesting stuff".  It's 
not going to be a spike at DC -- it's going to bleed out into adjoining 
frequency bins.  If there's something interesting (and small) in those 
bins, then it'll be lost in the noise generated by the DFT of the window.

You can easily end up with a frequency plot that's nothing more than the 
DFT of whatever window you chose, if you fail to remove the DC bias first.

Been there (when I had hair), done that, still remember...

-- 
My liberal friends think I'm a conservative kook.
My conservative friends think I'm a liberal kook.
Why am I not happy that they have found common ground?

Tim Wescott, Communications, Control, Circuits & Software
http://www.wescottdesign.com

Tim Wescott <tim@seemywebsite.com> wrote:
> On Fri, 05 Oct 2012 14:37:25 -0700, dbd wrote:
>> On Thursday, October 4, 2012 1:13:28 PM UTC-7, Tim Wescott wrote: ...

>>> So, having established your expertise by insult, please back up that
>>> statement!

(snip)
>> Your simple statement didn't include this qualification or the small DFT
>> size case glen discusses. (Or the limited resolution situation case
>> which becomes important if you include the big signal qualification.)
 
(bigger snip)

> Well, you seem to be missing what I was talking about.  If you apply a 
> _window_ to data with a DC bias, then your result after DFT is going to 
> be the DFT of the window, plus the DFT of the "interesting stuff".  It's 
> not going to be a spike at DC -- it's going to bleed out into adjoining 
> frequency bins.  If there's something interesting (and small) in those 
> bins, then it'll be lost in the noise generated by the DFT of the window.

Yes. But some windows will have only small components other than
at zero, and some signals will have enough non-zero frequency
components that you don't notice.

> You can easily end up with a frequency plot that's nothing more than the 
> DFT of whatever window you chose, if you fail to remove the DC bias first.

That can also happen.

Some time ago, I was thinking about how to explain Nyquist sampling
in the case of a finite length band limited signal. If both end 
points are zero, then it is easy to argue that it is made up
of sines of the appropriate frequencies up to the band limit.

If the end points aren't zero, then subtract a straight line through
the two end points, and you need two more values to reproduce
the original. The sines should need 2N-1 where N is the length
divided by the sampling period, truncated, so slightly more than
Nyquist would lead you to believe.

The Nyquist rate works for either infinite length or periodic signals.

-- glen

On Fri, 05 Oct 2012 19:33:07 -0500, Tim Wescott <tim@seemywebsite.com>
wrote:

>On Fri, 05 Oct 2012 14:37:25 -0700, dbd wrote:
>
>> On Thursday, October 4, 2012 1:13:28 PM UTC-7, Tim Wescott wrote: ...
>> 
>>> So, having established your expertise by insult, please back up that
>>> statement!
>> 
>> And what backup did you give your statement?
>> 
>> 
>>> Show how taking the DFT of a sample with a small signal embedded in a
>>> honkin' big DC offset, windowed, _doesn't_ have a huge contribution
>>> from the window!
>>> 
>>> 
>> Your simple statement didn't include this qualification or the small DFT
>> size case glen discusses. (Or the limited resolution situation case
>> which becomes important if you include the big signal qualification.)
>> 
>> The DFT is a linear operator. The mean of the time domain data is the DC
>> term in the frequency domain. Subtraction of this component from both
>> domains does not alter any term in the frequency domain but the DC term.
>> 
>>> ... particularly in that it is something that's being pointed out to an
>>> obviously rank beginner who clearly hasn't had much mileage at all with
>>> using the DFT for signal analysis.
>> 
>> You post a lot here and often usefully. If a stupid poster had posted
>> something stupid I might not comment. You know how to post coherently.
>> You didn't here. Exactly because of the OP's inexperience, I felt it
>> worth a quick kick.
>> 
>>> Tim Wescott
>> 
>> Dale B. Dalrymple
>
>Well, you seem to be missing what I was talking about.  If you apply a 
>_window_ to data with a DC bias, then your result after DFT is going to 
>be the DFT of the window, plus the DFT of the "interesting stuff". 

I'm struggling to understand the issue.   The DC bin is the same as
any other bin a DFT.   The output will be the DFT of the unwindowed
input convolved with the DFT of the window.   Bin zero is part of this
just like any other bin, and not any more or less than any other bin.

> It's 
>not going to be a spike at DC -- it's going to bleed out into adjoining 
>frequency bins.

Just like every other bin will, consistent with the DFT of the window
function.  

>  If there's something interesting (and small) in those 
>bins, then it'll be lost in the noise generated by the DFT of the window.

Just like any bin can swamp any other bin if the relative magnitudes
and sidelobes work out that way.

>You can easily end up with a frequency plot that's nothing more than the 
>DFT of whatever window you chose, if you fail to remove the DC bias first.

And any tone present can swamp adjacent tones if it's big enough and
they're small enough.  That's normal.

>Been there (when I had hair), done that, still remember...
>
>-- 
>My liberal friends think I'm a conservative kook.
>My conservative friends think I'm a liberal kook.
>Why am I not happy that they have found common ground?
>
>Tim Wescott, Communications, Control, Circuits & Software
>http://www.wescottdesign.com

Eric Jacobsen
Anchor Hill Communications
www.anchorhill.com

On Friday, October 5, 2012 5:33:08 PM UTC-7, Tim Wescott wrote:

> 
> Well, you seem to be missing what I was talking about.  If you apply a 
> 
> _window_ to data with a DC bias, then your result after DFT is going to 
> 
> be the DFT of the window, plus the DFT of the "interesting stuff".  It's 
> 
> not going to be a spike at DC -- it's going to bleed out into adjoining 
> 
> frequency bins.  If there's something interesting (and small) in those 
> 
> bins, then it'll be lost in the noise generated by the DFT of the window.
> 
> 
> 
> You can easily end up with a frequency plot that's nothing more than the 
> 
> DFT of whatever window you chose, if you fail to remove the DC bias first.
> 
> ...
> Tim Wescott, Communications, Control, Circuits & Software

The most commonly used window, the rectangular, does not bleed signal into adjacent bins. This is the window I intended in my last post (and should have stated explicitly). For any window length that is a positive integer multiple of the transform size, the rectangular window will have zeros for its response at the other frequency bins. Then the dft of the DC term is only in the DC bin and is all that is in the DC bin. Other length rectangular windows will not have this effect, for example, zero filling a data set with zeros to match a larger transform size.

Even with the rectangular window of the right size there are significant implementation details. The DC term might exceed the dynamic range of the dft implementation or force the other signal stuff down towards the quantization limits of the dft implementation. A frequent real world problem is that the big term is not exactly at DC, for example, a 60 Hz interference on high frequency samples with a window of much less than 16msec duration. These problems are not windows issues.

If you want to use a non-rectangular window in the very large true DC case, with the cosine-sum window family (including von Hann, Hamming, the Blackmans, etc.) you can use the dft DC coefficient to approximately estimate the correction to the adjacent dft coefficiets. Aside from quantization limitations of the impementation, windows do not produce "noise generated by the fft of the window". The effects are determinate, the dft of the window as has been suggested, which effects are zero for the rectangular window of appropriate lengths. The rectangular window can be viewed as the cosine-sum window where only 1 coefficient needs correction.

Dale B. Dalrymple

I have a question regarding this discussion. I have a signal for which I
have to take the DFT, but it is being multiplied by e^(j omega n), i.e.,
has a frequency offset. If I take the DFT and the period of omega doesn't
coincide with transform size, I have DFT leakage. I know omega but due to
some reason, I can't pre-correct it. What can we do in such a situation
(except windowing)?
Thank you.