Not a member?

# Discussion Groups | Comp.DSP | Purpose of Detrending signal before FFT?

There are 17 messages in this thread.

You are currently looking at messages 1 to .

Is this discussion worth a thumbs up?

+1

# Purpose of Detrending signal before FFT? - MBALOVER - 2012-10-01 19:59:00

```Hi all,

In code of an algorithm I see that the author often detrends signal before FFT. I search for it
in the Internet and the objective of the function is "detrend removes the mean value or
linear trend from a vector or matrix, usually for FFT processing".

Can you explain what is the effect of detrend and what if the data is not detrended before FTT?

Thank you.

```
______________________________

# Re: Purpose of Detrending signal before FFT? - Tim Wescott - 2012-10-01 20:08:00

```On Mon, 01 Oct 2012 16:59:54 -0700, MBALOVER wrote:

> Hi all,
>
> In code of an algorithm I see that the author often detrends signal
> before FFT. I search for it in the Internet and the objective of the
> function is "detrend removes the mean value or linear trend from a
> vector or matrix, usually for FFT processing".
>
> Can you explain what is the effect of detrend and what if the data is
> not detrended before FTT?
>
> Thank you.

Mathematically, what the FFT (more properly the discrete Fourier
transform -- the FFT is just a fast algorithm for performing the DFT)
extracts the components of a signal of finite length.

Generally, for signal analysis the FFT is used on signals of infinite
length.  So, when you chop up your signal into finite-length segments,
you are making an approximation.  Detrending the signal makes this
approximation better: the DFT acts as if it's signal "wraps around" from
the Nth data point to the first -- so if there's a big mismatch at that
point, the DFT will "see" signal components from the discontinuity.  But
the discontinuity isn't in the original signal -- you made it when you
whacked it up.  So it's up to you to remove it, in an effort to make your
approximation as good as possible.

Ultimately this is the same reason that the data is usually windowed,
although you're curing somewhat different problems with windowing vs.
detrending.

--
My liberal friends think I'm a conservative kook.
My conservative friends think I'm a liberal kook.
Why am I not happy that they have found common ground?

Tim Wescott, Communications, Control, Circuits & Software
http://www.wescottdesign.com
```
______________________________

# Re: Purpose of Detrending signal before FFT? - MBALOVER - 2012-10-01 20:21:00

```On Monday, October 1, 2012 5:08:13 PM UTC-7, Tim Wescott wrote:
> On Mon, 01 Oct 2012 16:59:54 -0700, MBALOVER wrote:
>
>
>
> > Hi all,
>
> >
>
> > In code of an algorithm I see that the author often detrends signal
>
> > before FFT. I search for it in the Internet and the objective of the
>
> > function is "detrend removes the mean value or linear trend from a
>
> > vector or matrix, usually for FFT processing".
>
> >
>
> > Can you explain what is the effect of detrend and what if the data is
>
> > not detrended before FTT?
>
> >
>
> > Thank you.
>
>
>
> Mathematically, what the FFT (more properly the discrete Fourier
>
> transform -- the FFT is just a fast algorithm for performing the DFT)
>
> extracts the components of a signal of finite length.
>
>
>
> Generally, for signal analysis the FFT is used on signals of infinite
>
> length.  So, when you chop up your signal into finite-length segments,
>
> you are making an approximation.  Detrending the signal makes this
>
> approximation better: the DFT acts as if it's signal "wraps around" from
>
> the Nth data point to the first -- so if there's a big mismatch at that
>
> point, the DFT will "see" signal components from the discontinuity.  But
>
> the discontinuity isn't in the original signal -- you made it when you
>
> whacked it up.  So it's up to you to remove it, in an effort to make your
>
> approximation as good as possible.
>
>
>
> Ultimately this is the same reason that the data is usually windowed,
>
> although you're curing somewhat different problems with windowing vs.
>
> detrending.
>
>
>
> --
>
> My liberal friends think I'm a conservative kook.
>
> My conservative friends think I'm a liberal kook.
>
> Why am I not happy that they have found common ground?
>
>
>
> Tim Wescott, Communications, Control, Circuits & Software
>
> http://www.wescottdesign.com

Thanks Tim,

So instead of detrending the signal, I can apply some kind of windows and will get similar
results. Is it correct?

```
______________________________

# Re: Purpose of Detrending signal before FFT? - glen herrmannsfeldt - 2012-10-01 20:41:00

```Tim Wescott <t...@seemywebsite.com> wrote:
> On Mon, 01 Oct 2012 16:59:54 -0700, MBALOVER wrote:

>> In code of an algorithm I see that the author often detrends signal
>> before FFT. I search for it in the Internet and the objective of the
>> function is "detrend removes the mean value or linear trend from a
>> vector or matrix, usually for FFT processing".

>> Can you explain what is the effect of detrend and what if the data is
>> not detrended before FTT?

(snip)
> Mathematically, what the FFT (more properly the discrete Fourier
> transform -- the FFT is just a fast algorithm for performing the DFT)
> extracts the components of a signal of finite length.

For DFT, it is periodic boundary conditions, so I like to think
of the signal as periodic instead of finite length.

For DST and DCT, finite length isn't so bad.

DST has zero for the boundary condition, DCT has zero derivative.

> Generally, for signal analysis the FFT is used on signals of infinite
> length.  So, when you chop up your signal into finite-length segments,
> you are making an approximation.  Detrending the signal makes this
> approximation better: the DFT acts as if it's signal "wraps around" from
> the Nth data point to the first -- so if there's a big mismatch at that
> point, the DFT will "see" signal components from the discontinuity.  But
> the discontinuity isn't in the original signal -- you made it when you
> whacked it up.  So it's up to you to remove it, in an effort to make your
> approximation as good as possible.

> Ultimately this is the same reason that the data is usually windowed,
> although you're curing somewhat different problems with windowing vs.
> detrending.

-- glen
```
______________________________

# Re: Purpose of Detrending signal before FFT? - Tim Wescott - 2012-10-01 20:44:00

```On Mon, 01 Oct 2012 17:21:21 -0700, MBALOVER wrote:

> On Monday, October 1, 2012 5:08:13 PM UTC-7, Tim Wescott wrote:
>> On Mon, 01 Oct 2012 16:59:54 -0700, MBALOVER wrote:
>>
>>
>>
>> > Hi all,
>>
>>
>> >
>> > In code of an algorithm I see that the author often detrends signal
>>
>> > before FFT. I search for it in the Internet and the objective of the
>>
>> > function is "detrend removes the mean value or linear trend from a
>>
>> > vector or matrix, usually for FFT processing".
>>
>>
>> >
>> > Can you explain what is the effect of detrend and what if the data is
>>
>> > not detrended before FTT?
>>
>>
>> >
>> > Thank you.
>>
>>
>>
>> Mathematically, what the FFT (more properly the discrete Fourier
>>
>> transform -- the FFT is just a fast algorithm for performing the DFT)
>>
>> extracts the components of a signal of finite length.
>>
>>
>>
>> Generally, for signal analysis the FFT is used on signals of infinite
>>
>> length.  So, when you chop up your signal into finite-length segments,
>>
>> you are making an approximation.  Detrending the signal makes this
>>
>> approximation better: the DFT acts as if it's signal "wraps around"
>> from
>>
>> the Nth data point to the first -- so if there's a big mismatch at that
>>
>> point, the DFT will "see" signal components from the discontinuity.
>> But
>>
>> the discontinuity isn't in the original signal -- you made it when you
>>
>> whacked it up.  So it's up to you to remove it, in an effort to make
>> your
>>
>> approximation as good as possible.
>>
>>
>>
>> Ultimately this is the same reason that the data is usually windowed,
>>
>> although you're curing somewhat different problems with windowing vs.
>>
>> detrending.
>>
>>
>>
>> --
>>
>> My liberal friends think I'm a conservative kook.
>>
>> My conservative friends think I'm a liberal kook.
>>
>> Why am I not happy that they have found common ground?
>>
>>
>>
>> Tim Wescott, Communications, Control, Circuits & Software
>>
>> http://www.wescottdesign.com
>
> Thanks Tim,
>
> So instead of detrending the signal, I can apply some kind of windows
> and will get similar results. Is it correct?

Nope.  Detrending and windowing are both ways to deal with the fact that
you're approximating an infinitely-long signal with a finite-length one,
but each has an effect and (generally) each must be applied.

If your data doesn't have any trends within the whacked-off portions then
you don't need detrending.  Similarly, if most of the signal energy is in
the high-frequency regions rather than low, you may be able to get away
without windowing.  In some oddball cases you're looking at data that is
very close to cyclic, in which case windowing is actually bad.

But nearly always, you need to detrend, then window.

--
My liberal friends think I'm a conservative kook.
My conservative friends think I'm a liberal kook.
Why am I not happy that they have found common ground?

Tim Wescott, Communications, Control, Circuits & Software
http://www.wescottdesign.com
```
______________________________

# Re: Purpose of Detrending signal before FFT? - dbd - 2012-10-02 00:49:00

```On Monday, October 1, 2012 4:59:55 PM UTC-7, MBALOVER wrote:
>...
> Can you explain what is the effect of detrend and what if the data is not detrended before
FTT?
>
>
>
> Thank you.

Removal of the mean is a good practice if you have instrumental data that has the potential for
a DC offset larger than the signals of interest. If the instumental artifact is a linear sweep,
that could be usefully removed as well. If the problem conponent does not exactly match the
ideal component subtracted, you may introduce large artifacts.

There may be statistical calculations that will not function as intended in the presence of a
large DC term. The DFT is not one of them.

Dale B. Dalrymple
```
______________________________

# Re: Purpose of Detrending signal before FFT? - Tim Wescott - 2012-10-02 12:13:00

```On Mon, 01 Oct 2012 21:49:01 -0700, dbd wrote:

> On Monday, October 1, 2012 4:59:55 PM UTC-7, MBALOVER wrote:
>>...
>> Can you explain what is the effect of detrend and what if the data is
>> not detrended before FTT?
>>
>>
>>
>> Thank you.
>
> Removal of the mean is a good practice if you have instrumental data
> that has the potential for a DC offset larger than the signals of
> interest. If the instumental artifact is a linear sweep, that could be
> usefully removed as well. If the problem conponent does not exactly
> match the ideal component subtracted, you may introduce large artifacts.
>
> There may be statistical calculations that will not function as intended
> in the presence of a large DC term. The DFT is not one of them.
>
> Dale B. Dalrymple

If you're windowing you need to remove the mean, or you just end up with
the DFT of the window.

--
Tim Wescott
Control system and signal processing consulting
www.wescottdesign.com
```
______________________________

# Re: Purpose of Detrending signal before FFT? - Benoit Leprettre - 2012-10-04 05:21:00

```Le mardi 2 octobre 2012 01:59:55 UTC+2, MBALOVER a écrit :
> Hi all,
>
>
>
> In code of an algorithm I see that the author often detrends signal before FFT. I search
for it in the Internet and the objective of the function is "detrend removes the mean value
or linear trend from a vector or matrix, usually for FFT processing".
>
> Can you explain what is the effect of detrend and what if the data is not detrended before
FTT?

Hi,

Removing artifacts caused by end-to-end discontinuities is one thing and detrending may, in some
case, contribute to solving that. However, the real thing to attenuate these artifacts is rather
windowing.

Detrending serves a different purpose. Suppose your trend in a large constant (DC component =
average). The result of your FFT will show a very large value at the zero frequency (DC).

This value in likely to "mask" other interesting features of your spectrum, because it
larger. It's annoying because you need to zoom in to see the other frequencies of interest. It's
not that the interesting information is not there, it's just that it is hidden by your DC. If
you want to remove it to concentrate on the "interesting" frequencies of your signal,
you just detrend the input signal by removing its mean.

Similarly, if you have a visible trend like a linear trend, that is, a slow variation of your
signal, then your spectrum will be "polluted" by its low frequency components the same
way as with a mere DC component. So, provided you know this trend is not significant for your
purpose, you may want to remove it beforehand, to better highlight other, higher-frequency, more

B.L.
```
______________________________

# Re: Purpose of Detrending signal before FFT? - dbd - 2012-10-04 15:25:00

```On Tuesday, October 2, 2012 9:13:07 AM UTC-7, Tim Wescott wrote:
...
>
> If you're windowing you need to remove the mean, or you just end up with
> the DFT of the window.

It's silly posts like this that make Vlad post the way he does.

> Tim Wescott
> Control system and signal processing consulting
except apparently, the DFT

Dale B. Dalrymple
```
______________________________

# Re: Purpose of Detrending signal before FFT? - Tim Wescott - 2012-10-04 16:13:00

```On Thu, 04 Oct 2012 12:25:10 -0700, dbd wrote:

> On Tuesday, October 2, 2012 9:13:07 AM UTC-7, Tim Wescott wrote: ...
>>
>> If you're windowing you need to remove the mean, or you just end up
>> with the DFT of the window.
>
> It's silly posts like this that make Vlad post the way he does.
>
>> Tim Wescott
>> Control system and signal processing consulting
>      except apparently, the DFT
>
> Dale B. Dalrymple

Excuse me!

statement!

Show how taking the DFT of a sample with a small signal embedded in a
honkin' big DC offset, windowed, _doesn't_ have a huge contribution from
the window!

Assuming that you can't do that, then please explain exactly how my --
true -- statement is "silly", particularly in that it is something that's
being pointed out to an obviously rank beginner who clearly hasn't had
much mileage at all with using the DFT for signal analysis.