Hi all, In code of an algorithm I see that the author often detrends signal before FFT. I search for it in the Internet and the objective of the function is "detrend removes the mean value or linear trend from a vector or matrix, usually for FFT processing". Can you explain what is the effect of detrend and what if the data is not detrended before FTT? Thank you.
Purpose of Detrending signal before FFT?
Started by ●October 1, 2012
Reply by ●October 1, 20122012-10-01
On Mon, 01 Oct 2012 16:59:54 -0700, MBALOVER wrote:> Hi all, > > In code of an algorithm I see that the author often detrends signal > before FFT. I search for it in the Internet and the objective of the > function is "detrend removes the mean value or linear trend from a > vector or matrix, usually for FFT processing". > > Can you explain what is the effect of detrend and what if the data is > not detrended before FTT? > > Thank you.Mathematically, what the FFT (more properly the discrete Fourier transform -- the FFT is just a fast algorithm for performing the DFT) extracts the components of a signal of finite length. Generally, for signal analysis the FFT is used on signals of infinite length. So, when you chop up your signal into finite-length segments, you are making an approximation. Detrending the signal makes this approximation better: the DFT acts as if it's signal "wraps around" from the Nth data point to the first -- so if there's a big mismatch at that point, the DFT will "see" signal components from the discontinuity. But the discontinuity isn't in the original signal -- you made it when you whacked it up. So it's up to you to remove it, in an effort to make your approximation as good as possible. Ultimately this is the same reason that the data is usually windowed, although you're curing somewhat different problems with windowing vs. detrending. -- My liberal friends think I'm a conservative kook. My conservative friends think I'm a liberal kook. Why am I not happy that they have found common ground? Tim Wescott, Communications, Control, Circuits & Software http://www.wescottdesign.com
Reply by ●October 1, 20122012-10-01
On Monday, October 1, 2012 5:08:13 PM UTC-7, Tim Wescott wrote:> On Mon, 01 Oct 2012 16:59:54 -0700, MBALOVER wrote: > > > > > Hi all, > > > > > > In code of an algorithm I see that the author often detrends signal > > > before FFT. I search for it in the Internet and the objective of the > > > function is "detrend removes the mean value or linear trend from a > > > vector or matrix, usually for FFT processing". > > > > > > Can you explain what is the effect of detrend and what if the data is > > > not detrended before FTT? > > > > > > Thank you. > > > > Mathematically, what the FFT (more properly the discrete Fourier > > transform -- the FFT is just a fast algorithm for performing the DFT) > > extracts the components of a signal of finite length. > > > > Generally, for signal analysis the FFT is used on signals of infinite > > length. So, when you chop up your signal into finite-length segments, > > you are making an approximation. Detrending the signal makes this > > approximation better: the DFT acts as if it's signal "wraps around" from > > the Nth data point to the first -- so if there's a big mismatch at that > > point, the DFT will "see" signal components from the discontinuity. But > > the discontinuity isn't in the original signal -- you made it when you > > whacked it up. So it's up to you to remove it, in an effort to make your > > approximation as good as possible. > > > > Ultimately this is the same reason that the data is usually windowed, > > although you're curing somewhat different problems with windowing vs. > > detrending. > > > > -- > > My liberal friends think I'm a conservative kook. > > My conservative friends think I'm a liberal kook. > > Why am I not happy that they have found common ground? > > > > Tim Wescott, Communications, Control, Circuits & Software > > http://www.wescottdesign.comThanks Tim, So instead of detrending the signal, I can apply some kind of windows and will get similar results. Is it correct?
Reply by ●October 1, 20122012-10-01
Tim Wescott <tim@seemywebsite.com> wrote:> On Mon, 01 Oct 2012 16:59:54 -0700, MBALOVER wrote:>> In code of an algorithm I see that the author often detrends signal >> before FFT. I search for it in the Internet and the objective of the >> function is "detrend removes the mean value or linear trend from a >> vector or matrix, usually for FFT processing".>> Can you explain what is the effect of detrend and what if the data is >> not detrended before FTT?(snip)> Mathematically, what the FFT (more properly the discrete Fourier > transform -- the FFT is just a fast algorithm for performing the DFT) > extracts the components of a signal of finite length.For DFT, it is periodic boundary conditions, so I like to think of the signal as periodic instead of finite length. For DST and DCT, finite length isn't so bad. DST has zero for the boundary condition, DCT has zero derivative.> Generally, for signal analysis the FFT is used on signals of infinite > length. So, when you chop up your signal into finite-length segments, > you are making an approximation. Detrending the signal makes this > approximation better: the DFT acts as if it's signal "wraps around" from > the Nth data point to the first -- so if there's a big mismatch at that > point, the DFT will "see" signal components from the discontinuity. But > the discontinuity isn't in the original signal -- you made it when you > whacked it up. So it's up to you to remove it, in an effort to make your > approximation as good as possible.> Ultimately this is the same reason that the data is usually windowed, > although you're curing somewhat different problems with windowing vs. > detrending.-- glen
Reply by ●October 1, 20122012-10-01
On Mon, 01 Oct 2012 17:21:21 -0700, MBALOVER wrote:> On Monday, October 1, 2012 5:08:13 PM UTC-7, Tim Wescott wrote: >> On Mon, 01 Oct 2012 16:59:54 -0700, MBALOVER wrote: >> >> >> >> > Hi all, >> >> >> > >> > In code of an algorithm I see that the author often detrends signal >> >> > before FFT. I search for it in the Internet and the objective of the >> >> > function is "detrend removes the mean value or linear trend from a >> >> > vector or matrix, usually for FFT processing". >> >> >> > >> > Can you explain what is the effect of detrend and what if the data is >> >> > not detrended before FTT? >> >> >> > >> > Thank you. >> >> >> >> Mathematically, what the FFT (more properly the discrete Fourier >> >> transform -- the FFT is just a fast algorithm for performing the DFT) >> >> extracts the components of a signal of finite length. >> >> >> >> Generally, for signal analysis the FFT is used on signals of infinite >> >> length. So, when you chop up your signal into finite-length segments, >> >> you are making an approximation. Detrending the signal makes this >> >> approximation better: the DFT acts as if it's signal "wraps around" >> from >> >> the Nth data point to the first -- so if there's a big mismatch at that >> >> point, the DFT will "see" signal components from the discontinuity. >> But >> >> the discontinuity isn't in the original signal -- you made it when you >> >> whacked it up. So it's up to you to remove it, in an effort to make >> your >> >> approximation as good as possible. >> >> >> >> Ultimately this is the same reason that the data is usually windowed, >> >> although you're curing somewhat different problems with windowing vs. >> >> detrending. >> >> >> >> -- >> >> My liberal friends think I'm a conservative kook. >> >> My conservative friends think I'm a liberal kook. >> >> Why am I not happy that they have found common ground? >> >> >> >> Tim Wescott, Communications, Control, Circuits & Software >> >> http://www.wescottdesign.com > > Thanks Tim, > > So instead of detrending the signal, I can apply some kind of windows > and will get similar results. Is it correct?Nope. Detrending and windowing are both ways to deal with the fact that you're approximating an infinitely-long signal with a finite-length one, but each has an effect and (generally) each must be applied. If your data doesn't have any trends within the whacked-off portions then you don't need detrending. Similarly, if most of the signal energy is in the high-frequency regions rather than low, you may be able to get away without windowing. In some oddball cases you're looking at data that is very close to cyclic, in which case windowing is actually bad. But nearly always, you need to detrend, then window. -- My liberal friends think I'm a conservative kook. My conservative friends think I'm a liberal kook. Why am I not happy that they have found common ground? Tim Wescott, Communications, Control, Circuits & Software http://www.wescottdesign.com
Reply by ●October 2, 20122012-10-02
On Monday, October 1, 2012 4:59:55 PM UTC-7, MBALOVER wrote:>... > Can you explain what is the effect of detrend and what if the data is not detrended before FTT? > > > > Thank you.Removal of the mean is a good practice if you have instrumental data that has the potential for a DC offset larger than the signals of interest. If the instumental artifact is a linear sweep, that could be usefully removed as well. If the problem conponent does not exactly match the ideal component subtracted, you may introduce large artifacts. There may be statistical calculations that will not function as intended in the presence of a large DC term. The DFT is not one of them. Dale B. Dalrymple
Reply by ●October 2, 20122012-10-02
On Mon, 01 Oct 2012 21:49:01 -0700, dbd wrote:> On Monday, October 1, 2012 4:59:55 PM UTC-7, MBALOVER wrote: >>... >> Can you explain what is the effect of detrend and what if the data is >> not detrended before FTT? >> >> >> >> Thank you. > > Removal of the mean is a good practice if you have instrumental data > that has the potential for a DC offset larger than the signals of > interest. If the instumental artifact is a linear sweep, that could be > usefully removed as well. If the problem conponent does not exactly > match the ideal component subtracted, you may introduce large artifacts. > > There may be statistical calculations that will not function as intended > in the presence of a large DC term. The DFT is not one of them. > > Dale B. DalrympleIf you're windowing you need to remove the mean, or you just end up with the DFT of the window. -- Tim Wescott Control system and signal processing consulting www.wescottdesign.com
Reply by ●October 4, 20122012-10-04
Le mardi 2 octobre 2012 01:59:55 UTC+2, MBALOVER a �crit�:> Hi all, > > > > In code of an algorithm I see that the author often detrends signal before FFT. I search for it in the Internet and the objective of the function is "detrend removes the mean value or linear trend from a vector or matrix, usually for FFT processing". > > Can you explain what is the effect of detrend and what if the data is not detrended before FTT?Hi, Removing artifacts caused by end-to-end discontinuities is one thing and detrending may, in some case, contribute to solving that. However, the real thing to attenuate these artifacts is rather windowing. Detrending serves a different purpose. Suppose your trend in a large constant (DC component = average). The result of your FFT will show a very large value at the zero frequency (DC). This value in likely to "mask" other interesting features of your spectrum, because it larger. It's annoying because you need to zoom in to see the other frequencies of interest. It's not that the interesting information is not there, it's just that it is hidden by your DC. If you want to remove it to concentrate on the "interesting" frequencies of your signal, you just detrend the input signal by removing its mean. Similarly, if you have a visible trend like a linear trend, that is, a slow variation of your signal, then your spectrum will be "polluted" by its low frequency components the same way as with a mere DC component. So, provided you know this trend is not significant for your purpose, you may want to remove it beforehand, to better highlight other, higher-frequency, more interesting features of your signal. B.L.
Reply by ●October 4, 20122012-10-04
On Tuesday, October 2, 2012 9:13:07 AM UTC-7, Tim Wescott wrote: ...> > If you're windowing you need to remove the mean, or you just end up with > the DFT of the window.It's silly posts like this that make Vlad post the way he does.> Tim Wescott > Control system and signal processing consultingexcept apparently, the DFT Dale B. Dalrymple
Reply by ●October 4, 20122012-10-04
On Thu, 04 Oct 2012 12:25:10 -0700, dbd wrote:> On Tuesday, October 2, 2012 9:13:07 AM UTC-7, Tim Wescott wrote: ... >> >> If you're windowing you need to remove the mean, or you just end up >> with the DFT of the window. > > It's silly posts like this that make Vlad post the way he does. > >> Tim Wescott >> Control system and signal processing consulting > except apparently, the DFT > > Dale B. DalrympleExcuse me! So, having established your expertise by insult, please back up that statement! Show how taking the DFT of a sample with a small signal embedded in a honkin' big DC offset, windowed, _doesn't_ have a huge contribution from the window! Assuming that you can't do that, then please explain exactly how my -- true -- statement is "silly", particularly in that it is something that's being pointed out to an obviously rank beginner who clearly hasn't had much mileage at all with using the DFT for signal analysis. Show your work. -- My liberal friends think I'm a conservative kook. My conservative friends think I'm a liberal kook. Why am I not happy that they have found common ground? Tim Wescott, Communications, Control, Circuits & Software http://www.wescottdesign.com
