Hi all,

In code of an algorithm I see that the author often detrends signal before FFT. I search for it
in the Internet and the objective of the function is "detrend removes the mean value or
linear trend from a vector or matrix, usually for FFT processing".

Can you explain what is the effect of detrend and what if the data is not detrended before FTT?

Thank you.

On Mon, 01 Oct 2012 16:59:54 -0700, MBALOVER wrote:

> Hi all,
> 
> In code of an algorithm I see that the author often detrends signal
> before FFT. I search for it in the Internet and the objective of the
> function is "detrend removes the mean value or linear trend from a
> vector or matrix, usually for FFT processing".
> 
> Can you explain what is the effect of detrend and what if the data is
> not detrended before FTT?
> 
> Thank you.

Mathematically, what the FFT (more properly the discrete Fourier 
transform -- the FFT is just a fast algorithm for performing the DFT) 
extracts the components of a signal of finite length.

Generally, for signal analysis the FFT is used on signals of infinite 
length.  So, when you chop up your signal into finite-length segments, 
you are making an approximation.  Detrending the signal makes this 
approximation better: the DFT acts as if it's signal "wraps around" from 
the Nth data point to the first -- so if there's a big mismatch at that 
point, the DFT will "see" signal components from the discontinuity.  But 
the discontinuity isn't in the original signal -- you made it when you 
whacked it up.  So it's up to you to remove it, in an effort to make your 
approximation as good as possible.

Ultimately this is the same reason that the data is usually windowed, 
although you're curing somewhat different problems with windowing vs. 
detrending.

-- 
My liberal friends think I'm a conservative kook.
My conservative friends think I'm a liberal kook.
Why am I not happy that they have found common ground?

Tim Wescott, Communications, Control, Circuits & Software
http://www.wescottdesign.com

On Monday, October 1, 2012 5:08:13 PM UTC-7, Tim Wescott wrote:
> On Mon, 01 Oct 2012 16:59:54 -0700, MBALOVER wrote:
> 
> 
> 
> > Hi all,
> 
> > 
> 
> > In code of an algorithm I see that the author often detrends signal
> 
> > before FFT. I search for it in the Internet and the objective of the
> 
> > function is "detrend removes the mean value or linear trend from a
> 
> > vector or matrix, usually for FFT processing".
> 
> > 
> 
> > Can you explain what is the effect of detrend and what if the data is
> 
> > not detrended before FTT?
> 
> > 
> 
> > Thank you.
> 
> 
> 
> Mathematically, what the FFT (more properly the discrete Fourier 
> 
> transform -- the FFT is just a fast algorithm for performing the DFT) 
> 
> extracts the components of a signal of finite length.
> 
> 
> 
> Generally, for signal analysis the FFT is used on signals of infinite 
> 
> length.  So, when you chop up your signal into finite-length segments, 
> 
> you are making an approximation.  Detrending the signal makes this 
> 
> approximation better: the DFT acts as if it's signal "wraps around" from 
> 
> the Nth data point to the first -- so if there's a big mismatch at that 
> 
> point, the DFT will "see" signal components from the discontinuity.  But 
> 
> the discontinuity isn't in the original signal -- you made it when you 
> 
> whacked it up.  So it's up to you to remove it, in an effort to make your 
> 
> approximation as good as possible.
> 
> 
> 
> Ultimately this is the same reason that the data is usually windowed, 
> 
> although you're curing somewhat different problems with windowing vs. 
> 
> detrending.
> 
> 
> 
> -- 
> 
> My liberal friends think I'm a conservative kook.
> 
> My conservative friends think I'm a liberal kook.
> 
> Why am I not happy that they have found common ground?
> 
> 
> 
> Tim Wescott, Communications, Control, Circuits & Software
> 
> http://www.wescottdesign.com

Thanks Tim,

So instead of detrending the signal, I can apply some kind of windows and will get similar
results. Is it correct?

Tim Wescott <t...@seemywebsite.com> wrote:
> On Mon, 01 Oct 2012 16:59:54 -0700, MBALOVER wrote:

>> In code of an algorithm I see that the author often detrends signal
>> before FFT. I search for it in the Internet and the objective of the
>> function is "detrend removes the mean value or linear trend from a
>> vector or matrix, usually for FFT processing".
 
>> Can you explain what is the effect of detrend and what if the data is
>> not detrended before FTT?

(snip)
> Mathematically, what the FFT (more properly the discrete Fourier 
> transform -- the FFT is just a fast algorithm for performing the DFT) 
> extracts the components of a signal of finite length.

For DFT, it is periodic boundary conditions, so I like to think
of the signal as periodic instead of finite length.

For DST and DCT, finite length isn't so bad.

DST has zero for the boundary condition, DCT has zero derivative.

> Generally, for signal analysis the FFT is used on signals of infinite 
> length.  So, when you chop up your signal into finite-length segments, 
> you are making an approximation.  Detrending the signal makes this 
> approximation better: the DFT acts as if it's signal "wraps around" from 
> the Nth data point to the first -- so if there's a big mismatch at that 
> point, the DFT will "see" signal components from the discontinuity.  But 
> the discontinuity isn't in the original signal -- you made it when you 
> whacked it up.  So it's up to you to remove it, in an effort to make your 
> approximation as good as possible.

> Ultimately this is the same reason that the data is usually windowed, 
> although you're curing somewhat different problems with windowing vs. 
> detrending.

-- glen

On Mon, 01 Oct 2012 17:21:21 -0700, MBALOVER wrote:

> On Monday, October 1, 2012 5:08:13 PM UTC-7, Tim Wescott wrote:
>> On Mon, 01 Oct 2012 16:59:54 -0700, MBALOVER wrote:
>> 
>> 
>> 
>> > Hi all,
>> 
>> 
>> > 
>> > In code of an algorithm I see that the author often detrends signal
>> 
>> > before FFT. I search for it in the Internet and the objective of the
>> 
>> > function is "detrend removes the mean value or linear trend from a
>> 
>> > vector or matrix, usually for FFT processing".
>> 
>> 
>> > 
>> > Can you explain what is the effect of detrend and what if the data is
>> 
>> > not detrended before FTT?
>> 
>> 
>> > 
>> > Thank you.
>> 
>> 
>> 
>> Mathematically, what the FFT (more properly the discrete Fourier
>> 
>> transform -- the FFT is just a fast algorithm for performing the DFT)
>> 
>> extracts the components of a signal of finite length.
>> 
>> 
>> 
>> Generally, for signal analysis the FFT is used on signals of infinite
>> 
>> length.  So, when you chop up your signal into finite-length segments,
>> 
>> you are making an approximation.  Detrending the signal makes this
>> 
>> approximation better: the DFT acts as if it's signal "wraps around"
>> from
>> 
>> the Nth data point to the first -- so if there's a big mismatch at that
>> 
>> point, the DFT will "see" signal components from the discontinuity. 
>> But
>> 
>> the discontinuity isn't in the original signal -- you made it when you
>> 
>> whacked it up.  So it's up to you to remove it, in an effort to make
>> your
>> 
>> approximation as good as possible.
>> 
>> 
>> 
>> Ultimately this is the same reason that the data is usually windowed,
>> 
>> although you're curing somewhat different problems with windowing vs.
>> 
>> detrending.
>> 
>> 
>> 
>> --
>> 
>> My liberal friends think I'm a conservative kook.
>> 
>> My conservative friends think I'm a liberal kook.
>> 
>> Why am I not happy that they have found common ground?
>> 
>> 
>> 
>> Tim Wescott, Communications, Control, Circuits & Software
>> 
>> http://www.wescottdesign.com
> 
> Thanks Tim,
> 
> So instead of detrending the signal, I can apply some kind of windows
> and will get similar results. Is it correct?

Nope.  Detrending and windowing are both ways to deal with the fact that 
you're approximating an infinitely-long signal with a finite-length one, 
but each has an effect and (generally) each must be applied.

If your data doesn't have any trends within the whacked-off portions then 
you don't need detrending.  Similarly, if most of the signal energy is in 
the high-frequency regions rather than low, you may be able to get away 
without windowing.  In some oddball cases you're looking at data that is 
very close to cyclic, in which case windowing is actually bad.

But nearly always, you need to detrend, then window.

-- 
My liberal friends think I'm a conservative kook.
My conservative friends think I'm a liberal kook.
Why am I not happy that they have found common ground?

Tim Wescott, Communications, Control, Circuits & Software
http://www.wescottdesign.com

On Monday, October 1, 2012 4:59:55 PM UTC-7, MBALOVER wrote:
>...
> Can you explain what is the effect of detrend and what if the data is not detrended before
FTT?
> 
> 
> 
> Thank you.

Removal of the mean is a good practice if you have instrumental data that has the potential for
a DC offset larger than the signals of interest. If the instumental artifact is a linear sweep,
that could be usefully removed as well. If the problem conponent does not exactly match the
ideal component subtracted, you may introduce large artifacts.

There may be statistical calculations that will not function as intended in the presence of a
large DC term. The DFT is not one of them.

Dale B. Dalrymple

On Mon, 01 Oct 2012 21:49:01 -0700, dbd wrote:

> On Monday, October 1, 2012 4:59:55 PM UTC-7, MBALOVER wrote:
>>...
>> Can you explain what is the effect of detrend and what if the data is
>> not detrended before FTT?
>> 
>> 
>> 
>> Thank you.
> 
> Removal of the mean is a good practice if you have instrumental data
> that has the potential for a DC offset larger than the signals of
> interest. If the instumental artifact is a linear sweep, that could be
> usefully removed as well. If the problem conponent does not exactly
> match the ideal component subtracted, you may introduce large artifacts.
> 
> There may be statistical calculations that will not function as intended
> in the presence of a large DC term. The DFT is not one of them.
> 
> Dale B. Dalrymple

If you're windowing you need to remove the mean, or you just end up with 
the DFT of the window.

-- 
Tim Wescott
Control system and signal processing consulting
www.wescottdesign.com

Le mardi 2 octobre 2012 01:59:55 UTC+2, MBALOVER a écrit :
> Hi all,
> 
> 
> 
> In code of an algorithm I see that the author often detrends signal before FFT. I search
for it in the Internet and the objective of the function is "detrend removes the mean value
or linear trend from a vector or matrix, usually for FFT processing".
>
> Can you explain what is the effect of detrend and what if the data is not detrended before
FTT?

Hi,

Removing artifacts caused by end-to-end discontinuities is one thing and detrending may, in some
case, contribute to solving that. However, the real thing to attenuate these artifacts is rather
windowing.

Detrending serves a different purpose. Suppose your trend in a large constant (DC component =
average). The result of your FFT will show a very large value at the zero frequency (DC).

This value in likely to "mask" other interesting features of your spectrum, because it
larger. It's annoying because you need to zoom in to see the other frequencies of interest. It's
not that the interesting information is not there, it's just that it is hidden by your DC. If
you want to remove it to concentrate on the "interesting" frequencies of your signal,
you just detrend the input signal by removing its mean.

Similarly, if you have a visible trend like a linear trend, that is, a slow variation of your
signal, then your spectrum will be "polluted" by its low frequency components the same
way as with a mere DC component. So, provided you know this trend is not significant for your
purpose, you may want to remove it beforehand, to better highlight other, higher-frequency, more
interesting features of your signal.

B.L.

On Tuesday, October 2, 2012 9:13:07 AM UTC-7, Tim Wescott wrote:
...
> 
> If you're windowing you need to remove the mean, or you just end up with 
> the DFT of the window.

It's silly posts like this that make Vlad post the way he does.

> Tim Wescott
> Control system and signal processing consulting
     except apparently, the DFT

Dale B. Dalrymple

On Thu, 04 Oct 2012 12:25:10 -0700, dbd wrote:

> On Tuesday, October 2, 2012 9:13:07 AM UTC-7, Tim Wescott wrote: ...
>> 
>> If you're windowing you need to remove the mean, or you just end up
>> with the DFT of the window.
> 
> It's silly posts like this that make Vlad post the way he does.
> 
>> Tim Wescott
>> Control system and signal processing consulting
>      except apparently, the DFT
> 
> Dale B. Dalrymple

Excuse me!

So, having established your expertise by insult, please back up that 
statement!

Show how taking the DFT of a sample with a small signal embedded in a 
honkin' big DC offset, windowed, _doesn't_ have a huge contribution from 
the window!

Assuming that you can't do that, then please explain exactly how my -- 
true -- statement is "silly", particularly in that it is something that's 
being pointed out to an obviously rank beginner who clearly hasn't had 
much mileage at all with using the DFT for signal analysis.

Show your work.

-- 
My liberal friends think I'm a conservative kook.
My conservative friends think I'm a liberal kook.
Why am I not happy that they have found common ground?

Tim Wescott, Communications, Control, Circuits & Software
http://www.wescottdesign.com