Frequency domain filtering (rectangular window question)

Jerry Avins wrote:
> robert bristow-johnson wrote:
> > you cannot have any arbitrary frequency response and do "frequency
> > domain filtering" of a continuous stream of data or a very long file.

I've heard of people doing single ffts on files much to long to
fit in a computers main memory.  I think, in the old days, they
did this using multiple passes between tape drives.

> Hell, man, you can't even have an arbitrary frequency response even if
> the signal fits in the buffer. All one can do is pick the response of
> bin centers. Between those frequencies, the response can go wild.

Under what circumstances do frequencies between bin centers have any
meaning if the signal and its interpretation have no existance
(continuity constraints, repeatability, etc.) outside of the buffer?

I usually interpret certain piles of fft'd frequencies as artifacts of
windowing what is in fact a single non-bin frequency because I assume
that the waveform is continuous and repetitive outside the buffer
under analysis in some manner not related to an integral (sub)multiple
of the buffer size.  Without this assumption, is seems like between-bin
frequencies might themselves just be considered as representational
artifacts.


IMHO. YMMV.
-- 
rhn A.T nicholson d.O.t C-o-M

rhnlogic@yahoo.com wrote:
> Jerry Avins wrote:
> 
>>robert bristow-johnson wrote:
>>
>>>you cannot have any arbitrary frequency response and do "frequency
>>>domain filtering" of a continuous stream of data or a very long file.
> 
> 
> I've heard of people doing single ffts on files much to long to
> fit in a computers main memory.  I think, in the old days, they
> did this using multiple passes between tape drives.
> 
> 
>>Hell, man, you can't even have an arbitrary frequency response even if
>>the signal fits in the buffer. All one can do is pick the response of
>>bin centers. Between those frequencies, the response can go wild.
> 
> 
> Under what circumstances do frequencies between bin centers have any
> meaning if the signal and its interpretation have no existance
> (continuity constraints, repeatability, etc.) outside of the buffer?
> 
> I usually interpret certain piles of fft'd frequencies as artifacts of
> windowing what is in fact a single non-bin frequency because I assume
> that the waveform is continuous and repetitive outside the buffer
> under analysis in some manner not related to an integral (sub)multiple
> of the buffer size.  Without this assumption, is seems like between-bin
> frequencies might themselves just be considered as representational
> artifacts.

I don't get what you mean by artifact. Frequency response is a matter of 
gain vs. frequency. However much of an input frequency comes out is the 
filter's response. If it comes out as another frequency, the filter 
isn't linear.

Jerry
-- 
Engineering is the art of making what you want from things you can get.
�����������������������������������������������������������������������

Jerry Avins wrote:
> rhnlogic@yahoo.com wrote:
> > Jerry Avins wrote:
> >
> >>robert bristow-johnson wrote:
> >>
> >>>you cannot have any arbitrary frequency response and do "frequency
> >>>domain filtering" of a continuous stream of data or a very long file.
> >
> >
> > I've heard of people doing single ffts on files much to long to
> > fit in a computers main memory.  I think, in the old days, they
> > did this using multiple passes between tape drives.
> >
> >
> >>Hell, man, you can't even have an arbitrary frequency response even if
> >>the signal fits in the buffer. All one can do is pick the response of
> >>bin centers. Between those frequencies, the response can go wild.
> >
> >
> > Under what circumstances do frequencies between bin centers have any
> > meaning if the signal and its interpretation have no existance
> > (continuity constraints, repeatability, etc.) outside of the buffer?
> >
> > I usually interpret certain piles of fft'd frequencies as artifacts of
> > windowing what is in fact a single non-bin frequency because I assume
> > that the waveform is continuous and repetitive outside the buffer
> > under analysis in some manner not related to an integral (sub)multiple
> > of the buffer size.  Without this assumption, is seems like between-bin
> > frequencies might themselves just be considered as representational
> > artifacts.
>
> I don't get what you mean by artifact. Frequency response is a matter of
> gain vs. frequency. However much of an input frequency comes out is the
> filter's response. If it comes out as another frequency, the filter
> isn't linear.

What do you mean by the "frequency" of something in an array of samples
in a buffer?  If you convert everything into a complete discrete FT
representation, then there exists nothing between the bins to filter.
Whatever is in the bins can completely reconstruct the signal, given
the non-existance of any constraints on what happens outside the
buffer window (continuous, discontinuous, etc.)  So is the "response"
to any frequencies between the bins meaningful in the non-continuous
single buffer case?


IMHO. YMMV.
-- 
rhn

rhnlogic@yahoo.com wrote:
> Jerry Avins wrote:
> 
>>rhnlogic@yahoo.com wrote:
>>
>>>Jerry Avins wrote:
>>>
>>>
>>>>robert bristow-johnson wrote:
>>>>
>>>>
>>>>>you cannot have any arbitrary frequency response and do "frequency
>>>>>domain filtering" of a continuous stream of data or a very long file.
>>>
>>>
>>>I've heard of people doing single ffts on files much to long to
>>>fit in a computers main memory.  I think, in the old days, they
>>>did this using multiple passes between tape drives.
>>>
>>>
>>>
>>>>Hell, man, you can't even have an arbitrary frequency response even if
>>>>the signal fits in the buffer. All one can do is pick the response of
>>>>bin centers. Between those frequencies, the response can go wild.
>>>
>>>
>>>Under what circumstances do frequencies between bin centers have any
>>>meaning if the signal and its interpretation have no existance
>>>(continuity constraints, repeatability, etc.) outside of the buffer?
>>>
>>>I usually interpret certain piles of fft'd frequencies as artifacts of
>>>windowing what is in fact a single non-bin frequency because I assume
>>>that the waveform is continuous and repetitive outside the buffer
>>>under analysis in some manner not related to an integral (sub)multiple
>>>of the buffer size.  Without this assumption, is seems like between-bin
>>>frequencies might themselves just be considered as representational
>>>artifacts.
>>
>>I don't get what you mean by artifact. Frequency response is a matter of
>>gain vs. frequency. However much of an input frequency comes out is the
>>filter's response. If it comes out as another frequency, the filter
>>isn't linear.
> 
> 
> What do you mean by the "frequency" of something in an array of samples
> in a buffer?  If you convert everything into a complete discrete FT
> representation, then there exists nothing between the bins to filter.
> Whatever is in the bins can completely reconstruct the signal, given
> the non-existance of any constraints on what happens outside the
> buffer window (continuous, discontinuous, etc.)  So is the "response"
> to any frequencies between the bins meaningful in the non-continuous
> single buffer case?

If you leave it in the buffer, then no. There is nothing "between" the 
bins of an FT. But when you do an IFT, and interpret the results as 
samples of a continuous signal (or reconstruct that continuous signal) 
then all frequencies may be present. You have only indirect control of 
the frequencies between the FT's bins, and that control is exercised by 
the abruptness of the transitions specified in the frequency domain.

Jerry

P.S. IMHOs at ten paces. My opinion is humbler than yours. Nyaah!
-- 
Engineering is the art of making what you want from things you can get.
�����������������������������������������������������������������������

So let's see if I am getting this.  There is information between the bins
that I can't see in the freq. domain and that doesn't become visible until
an IFFT to go back into the time domain.  So by creating a "brick wall"
filter (as someone here refered to it) in the freq. domain, I am not able
to control the frequency response near the edge of the "brick wall".  The
frequencies that lie between the edge of my "brick wall" and the open
space beside it are not defined in the freq. domain (bins are too far
apart) so they could be present when I IFFT back to look at my filtered
waveform.  Am I reading/understanding correctly?  This still seems bizarre
to me as I don't really understand how we can have information "between"
the bins.  Doesn't the stuff "between" just get lumped in with the closest
bin, so that there is NO space between bins?

It seems to me that, if the above is true, I could just define my freq.
domain bins to be really closely spaced, that way there is a narrower band
of frequencies that can lie between my bins.  I can do this be either a)
padding the FFT so that there is a finer resolution (i.e., smaller freq.
bins) or b) sampling faster (which then increases the length of the FFT if
I sample for the same length of time).  That brings up another question:
how much faster should I sample than the frequencies I am interested in? 
I know I need to sample atleast 2X faster (Nyquist freq.), but is there a
better relationship/ratio for sampling freq. to freq. of interest?  What's
the DSP community rule of thumb for this?

I currently use 32786 or 65536 samples at a sampling rate of 10 MS/s (~3.3
or 6.5 ms long signals in time) and am intersted in frequencies 50 to 500
kHz.  When I do the FFT's I use the same number of FFT points as there are
samples in my signal (i.e, 32788 or 65536).  Should I be doing something
differently?

Thanks for all the comments.

		
This message was sent using the Comp.DSP web interface on
www.DSPRelated.com

tjuii wrote:
> So let's see if I am getting this.  There is information between the bins
> that I can't see in the freq. domain and that doesn't become visible until
> an IFFT to go back into the time domain.  So by creating a "brick wall"
> filter (as someone here refered to it) in the freq. domain, I am not able
> to control the frequency response near the edge of the "brick wall".  The
> frequencies that lie between the edge of my "brick wall" and the open
> space beside it are not defined in the freq. domain (bins are too far
> apart) so they could be present when I IFFT back to look at my filtered
> waveform.  Am I reading/understanding correctly?  This still seems bizarre
> to me as I don't really understand how we can have information "between"
> the bins.  Doesn't the stuff "between" just get lumped in with the closest
> bin, so that there is NO space between bins?

Supposing that no frequencies can be between the bin centers is like 
supposing that there is no time signal between ADC samples. The FT is a 
sampling of the continuous frequency response. If it were otherwise, 
your filter would turn a continuous sweep into a series of discrete 
frequencies.

> It seems to me that, if the above is true, I could just define my freq.
> domain bins to be really closely spaced, that way there is a narrower band
> of frequencies that can lie between my bins.  I can do this be either a)
> padding the FFT so that there is a finer resolution (i.e., smaller freq.
> bins) or b) sampling faster (which then increases the length of the FFT if
> I sample for the same length of time).  That brings up another question:
> how much faster should I sample than the frequencies I am interested in? 
> I know I need to sample atleast 2X faster (Nyquist freq.), but is there a
> better relationship/ratio for sampling freq. to freq. of interest?  What's
> the DSP community rule of thumb for this?

You can put the peaks and dips of the horrible frequency response closer 
together and move them closer to the filter's transition, but you can't 
reduce their amplitude. (See "Gibbs phenomenon.) Keep in mind that there 
is nothing you can do in the frequency domain that can't also be done in 
the time domain. If you come across a "method" that seems to contradict 
that, be assured it won't work.

> I currently use 32786 or 65536 samples at a sampling rate of 10 MS/s (~3.3
> or 6.5 ms long signals in time) and am intersted in frequencies 50 to 500
> kHz.  When I do the FFT's I use the same number of FFT points as there are
> samples in my signal (i.e, 32788 or 65536).  Should I be doing something
> differently?


Unless all the frequencies represented by the time samples have an 
integer number of cycles, there will be a discontinuity in the cyclic 
representation. Repeating the same data end to end FFTing that gives the 
same output (except with finer spacing) so you can visualize the 
discontinuity that way. The effect of it scatters energy thoughout the 
result, affecting most those bins near real signals. A tapering window 
reduces the effect but smears all the bins a bit. Different windows for 
different purposes. Google for them. (Hamming, Blackman, Nuttall, more.)
The same applies to IFFTs. Imposed discontinuities cause artifacts.

Jerry
-- 
Engineering is the art of making what you want from things you can get.
�����������������������������������������������������������������������

So let's see if I am getting this.  There is information between the bins
that I can't see in the freq. domain and that doesn't become visible until
an IFFT to go back into the time domain.  So by creating a "brick wall"
filter (as someone here refered to it) in the freq. domain, I am not able
to control the frequency response near the edge of the "brick wall".  The
frequencies that lie between the edge of my "brick wall" and the open
space beside it are not defined in the freq. domain (bins are too far
apart) so they could be present when I IFFT back to look at my filtered
waveform.  Am I reading/understanding correctly?  This still seems bizarre
to me as I don't really understand how we can have information "between"
the bins.  Doesn't the stuff "between" just get lumped in with the closest
bin, so that there is NO space between bins?

It seems to me that, if the above is true, I could just define my freq.
domain bins to be really closely spaced, that way there is a narrower band
of frequencies that can lie between my bins.  I can do this be either a)
padding the FFT so that there is a finer resolution (i.e., smaller freq.
bins) or b) sampling faster (which then increases the length of the FFT if
I sample for the same length of time).  That brings up another question:
how much faster should I sample than the frequencies I am interested in? 
I know I need to sample atleast 2X faster (Nyquist freq.), but is there a
better relationship/ratio for sampling freq. to freq. of interest?  What's
the DSP community rule of thumb for this?

I currently use 32786 or 65536 samples at a sampling rate of 10 MS/s (~3.3
or 6.5 ms long signals in time) and am intersted in frequencies 50 to 500
kHz.  When I do the FFT's I use the same number of FFT points as there are
samples in my signal (i.e, 32788 or 65536).  Should I be doing something
differently?

Thanks for all the comments.

		
This message was sent using the Comp.DSP web interface on
www.DSPRelated.com

Jerry Avins wrote:
This still seems bizarre
> > to me as I don't really understand how we can have information "between"
> > the bins.  Doesn't the stuff "between" just get lumped in with the
> > closest bin, so that there is NO space between bins?

Any signal "between the bins" gets smeared out among all the other
bins, not just the closest bin.  Try it: fft a single sine wave
with a frequency that is between the fft bins.  Also note the phases
of the resulting fft on opposite sides of your center frequency.

> > It seems to me that, if the above is true, I could just define my freq.
> > domain bins to be really closely spaced, that way there is a narrower band
> > of frequencies that can lie between my bins.  I can do this be either a)
> > padding the FFT so that there is a finer resolution (i.e., smaller freq.
> > bins) or b) sampling faster (which then increases the length of the FFT if
> > I sample for the same length of time). ...
>
> You can put the peaks and dips of the horrible frequency response closer
> together and move them closer to the filter's transition, but you can't
> reduce their amplitude. (See "Gibbs phenomenon.)

Hmmm... Are you sure?  I thought that if one oversamples, and then
uses the longer fft (representing the same amount of absolute time)
for a longer filter with softer transistions between the original
"brick wall" frequencies (but still keeping the original frequency
filter values also in the shorter filter at either 1.0 or 0) then the
amplitude of frequency response wiggles between those original
frequencies can be made smoother in the padded or oversampled case.


IMHO. YMMV.
-- 
rhn A.T. nicholson d.O.t C-o-M

rhnlogic@yahoo.com wrote:
> Jerry Avins wrote:

   ...

>>You can put the peaks and dips of the horrible frequency response closer
>>together and move them closer to the filter's transition, but you can't
>>reduce their amplitude. (See "Gibbs phenomenon.)
> 
> 
> Hmmm... Are you sure?  I thought that if one oversamples, and then
> uses the longer fft (representing the same amount of absolute time)
> for a longer filter with softer transistions between the original
> "brick wall" frequencies (but still keeping the original frequency
> filter values also in the shorter filter at either 1.0 or 0) then the
> amplitude of frequency response wiggles between those original
> frequencies can be made smoother in the padded or oversampled case.

An FFT brick wall isn't really. The mask can be zero at one frequency 
and zero at the next, but you can claim that the response ramps from one 
value to the other over the interval of one bin. The closer the bins, 
the steeper the ramp. The dips and peaks (or is that "doodles") get 
closer together when the bins get smaller, but they stay the same size.

A true "Brick wall" is infinitely long. Without a window, even that 
rings. (OK: all bets are off when passing to infinity. Infinity - 1 
rings. :-)

Displaying the oscillations on a scale that varied inversely with the 
number of term in the series shows that by the time 7 or so terms are 
included, the asymptotic behavior is very nearly reached. The "Expanded 
Reconstructions" in http://tinyurl.com/cmebr show that.

Jerry
-- 
Engineering is the art of making what you want from things you can get.
�����������������������������������������������������������������������

tjuii wrote:
>  This still seems bizarre
> to me as I don't really understand how we can have information "between"
> the bins.  Doesn't the stuff "between" just get lumped in with the closest
> bin, so that there is NO space between bins?

Consider that the DFT gives you discrete samples of an 
underlying continuous function of frequency.  It is that 
continuous function that lies between the samples.  That the 
function is fully specified by the samples (there is only 
one function that connects them within the band limiting 
condition) does not negate its existence.

"Bin" is a terrible misnomer for frequency domain samples 
which continues to persist for some reason.

> 
> It seems to me that, if the above is true, I could just define my freq.
> domain bins to be really closely spaced, that way there is a narrower band
> of frequencies that can lie between my bins.  

Right, and in so doing you are allowing for longer and 
longer time domain functions when you IFFT which give you 
more and more control over the detail of the continuous 
underlying function of frequency.

> I can do this be either a)
> padding the FFT so that there is a finer resolution (i.e., smaller freq.
> bins) 

Bingo!  I wouldn't call it padding, though.  You are just 
specifying more and more detail by making the frequency 
domain specification finer (and thus longer) which will 
result in longer time domain functions to realize that 
specification.  You still have little control over the 
continuous function that lies between the denser samples and 
can still see signifigant oscilation and deviation there.

If you are specifying your filter in the frequency domain 
what are you doing about the phase portion of that 
specification?


Bob
-- 

"Things should be described as simply as possible, but no 
simpler."

                                              A. Einstein