Reply by MartinC December 11, 20062006-12-11
>MartinC wrote: >> >Hi Martin, >> > >> >> I don't suppose I could get something for nothing by transforming to
the
>> frequency domain, filtering there, then transorming back to the time >> domain, could I? I think I'd still need the next block so that I
could
>> overlap appropriately in the frequency domain and leave the gain >> unchanged. True? > >You'd either have to buffer and overlap add/save, or you will end >up with a similar problem: either the filter will include zero padding >at the ends for a padded fft, or will filter in data from the opposite >end of the buffer if you don't pad, due the circular effects of >filtering that way.
Thanks Ron, Isn't there the problem as well that unless I use a rectangular window before I fft that I'll also need to undo the effect of that window on gain when I ifft? If the window rolls off the timeseries values to zero, I don't know that I can even do that without overlapping, can I? Martin
> >Another possibility would be to asymmetrically filter from the >left, then identically filter from the right, then blend the two >filter outputs using a ramp of some sort. You will probably >still end up with boundry artifacts due to phase differences. > > >IMHO. YMMV. >-- >rhn A.T nicholson d.0.t C-o-M > >
Reply by Ron N. December 10, 20062006-12-10
MartinC wrote:
> >Hi Martin, > > > >I think that your change in filter coefficient sets results in a time > >varying change in frequency response that results in a modulation of > >the data.
...
> I guess I'll need to bite the bullet one way or another. The problem > isn't really a lack of storage but rather working within the constraints > of the legacy code. Life would be a lot easier if I could do all the > necessary processing using just the block of data I have in hand at any > given time. > > I don't suppose I could get something for nothing by transforming to the > frequency domain, filtering there, then transorming back to the time > domain, could I? I think I'd still need the next block so that I could > overlap appropriately in the frequency domain and leave the gain > unchanged. True?
You'd either have to buffer and overlap add/save, or you will end up with a similar problem: either the filter will include zero padding at the ends for a padded fft, or will filter in data from the opposite end of the buffer if you don't pad, due the circular effects of filtering that way. Another possibility would be to asymmetrically filter from the left, then identically filter from the right, then blend the two filter outputs using a ramp of some sort. You will probably still end up with boundry artifacts due to phase differences. IMHO. YMMV. -- rhn A.T nicholson d.0.t C-o-M
Reply by MartinC December 10, 20062006-12-10
>Hi Martin, > >I think that your change in filter coefficient sets results in a time >varying change in frequency response that results in a modulation of >the data. > >The MATLAB 'filter' function used with A = 1.0 for a FIR filter >implementation and using the Zf and Zi parameters is an example of the >minimum size of storage to just read the data in once. try 'help >filter' in Matlab if you have it. > >If you can't afford to store this much data per channel you can avoid >storage by reading in a block of data of (blocksize + filterlength - 1) >size every blocksize samples and just running the normal length filter >to produce the correct number of outputs per block. This trades off a >little data bandwidth for maintaining data storage between filter >operations. > >Dale Dalrymple >http://dbdimages.com > >
Thanks Dale. Hey we could use a guy like you where I work! I guess I'll need to bite the bullet one way or another. The problem isn't really a lack of storage but rather working within the constraints of the legacy code. Life would be a lot easier if I could do all the necessary processing using just the block of data I have in hand at any given time. I don't suppose I could get something for nothing by transforming to the frequency domain, filtering there, then transorming back to the time domain, could I? I think I'd still need the next block so that I could overlap appropriately in the frequency domain and leave the gain unchanged. True? Good to hear from you again. I'll drop you a note a little later. Martin
Reply by dbd December 9, 20062006-12-09
Hi Martin,

I think that your change in filter coefficient sets results in a time
varying change in frequency response that results in a modulation of
the data.

The MATLAB 'filter' function used with A = 1.0 for a FIR filter
implementation and using the Zf and Zi parameters is an example of the
minimum size of storage to just read the data in once. try 'help
filter' in Matlab if you have it.

If you can't afford to store this much data per channel you can avoid
storage by reading in a block of data of (blocksize + filterlength - 1)
size every blocksize samples and just running the normal length filter
to produce the correct number of outputs per block. This trades off a
little data bandwidth for maintaining data storage between filter
operations.

Dale Dalrymple
http://dbdimages.com

Reply by Rune Allnor December 8, 20062006-12-08
MartinC skrev:
> > > >I knew I had missed something in this post. Your comment about the > >"blindfolds on edge" made me think about vertical stripes, but maybe > >you are talking about horizontal bands. > > > >If the latter, I suspect you see the effects of applying two different > >filters to the sequence; or possibly imperfections at the joints > >between consecutive buffers. > > > >Rune > > > > > > You were correct the first time, Rune. The problem is very narrow > vertical banding. The spacing is on the order of 1.6, 3.2, 6.4 Hz (don't > hold me to those values exactly, I'd need to go back to the lab to do more > checking) and at first blush those numbers appear to track the replay rate.
OK. Sounds like you should be looking for zeros in the filter transfer function, or poor pre-plotting interpolation. In a different post you said you trust the plotting software. Could there be some spectrum interpolation step between the filter and the plotter in the previous versions of the SW, that you have not included? Rune
Reply by Rune Allnor December 8, 20062006-12-08
MartinC skrev:
> > > >I knew I had missed something in this post. Your comment about the > >"blindfolds on edge" made me think about vertical stripes, but maybe > >you are talking about horizontal bands. > > > >If the latter, I suspect you see the effects of applying two different > >filters to the sequence; or possibly imperfections at the joints > >between consecutive buffers. > > > >Rune > > > > > > You were correct the first time, Rune. The problem is very narrow > vertical banding. The spacing is on the order of 1.6, 3.2, 6.4 Hz (don't > hold me to those values exactly, I'd need to go back to the lab to do more > checking) and at first blush those numbers appear to track the replay rate.
OK. Sounds like you should be looking for zeros in the filter transfer function, or poor pre-plotting interpolation. In a different post you said you trust the plotting software. Could there be some spectrum interpolation step between the filter and the plotter in the previous versions of the SW, that you have not included? Rune
Reply by Rune Allnor December 8, 20062006-12-08
MartinC skrev:
> > > >I knew I had missed something in this post. Your comment about the > >"blindfolds on edge" made me think about vertical stripes, but maybe > >you are talking about horizontal bands. > > > >If the latter, I suspect you see the effects of applying two different > >filters to the sequence; or possibly imperfections at the joints > >between consecutive buffers. > > > >Rune > > > > > > You were correct the first time, Rune. The problem is very narrow > vertical banding. The spacing is on the order of 1.6, 3.2, 6.4 Hz (don't > hold me to those values exactly, I'd need to go back to the lab to do more > checking) and at first blush those numbers appear to track the replay rate. > The entire frequency range we capture is variable, but ~2560 Hz > (basebanded) would be the limit at a replay rate of 8.
OK. Seems you would be looking for zeros in the filter transfer functions, then. Or could there be some pre-plotter spectrum interpolation that is missing? Rune
Reply by MartinC December 8, 20062006-12-08
> >I knew I had missed something in this post. Your comment about the >"blindfolds on edge" made me think about vertical stripes, but maybe >you are talking about horizontal bands. > >If the latter, I suspect you see the effects of applying two different >filters to the sequence; or possibly imperfections at the joints >between consecutive buffers. > >Rune > >
You were correct the first time, Rune. The problem is very narrow vertical banding. The spacing is on the order of 1.6, 3.2, 6.4 Hz (don't hold me to those values exactly, I'd need to go back to the lab to do more checking) and at first blush those numbers appear to track the replay rate. The entire frequency range we capture is variable, but ~2560 Hz (basebanded) would be the limit at a replay rate of 8. As to your point about "why not just go ahead and buffer", well, given my druthers, I would do that. Martin
Reply by Rune Allnor December 8, 20062006-12-08
MartinC skrev:
> I apply the long > filter to the input data, save the result, shift the data by the decimation > factor and repeat. I do this as long as the tail of the long filter does > not go beyond the block of input data. Then, from the place where the > next dot product would be formed and from then on I apply the short filter > in the same manner until it too would otherwise go beyond the block of > input data.
...
> After further processing I display the output in a LOFAR gram (a waterfall > of frequency (horizontal) vs. time (vertical)) and I see this striping... > like blinds turned on edge.
I knew I had missed something in this post. Your comment about the "blindfolds on edge" made me think about vertical stripes, but maybe you are talking about horizontal bands. If the latter, I suspect you see the effects of applying two different filters to the sequence; or possibly imperfections at the joints between consecutive buffers. Rune
Reply by Rune Allnor December 8, 20062006-12-08
MartinC skrev:
> > > >MartinC skrev: > > > >> After further processing I display the output in a LOFAR gram (a > waterfall > >> of frequency (horizontal) vs. time (vertical)) and I see this > striping... > >> like blinds turned on edge. Is this because there's a flaw in my > 'clever' > >> method or do you think there is some other source for this? > > > >Start with the easy stuff first: What type of data do you process? > >How do you plot the results? Have you tested the plotting routine > >with other data? > > > >Rune > > > > > > Hi Rune, > > The system accepts data from multiple sources, both analog and digital. > My starting point was to look at the case of 32 bit wave files so that I > could more easily control the inputs and make them repeatable, get that > working, then integrate the code into various places in the system. > > The system has been around in one form or another for many, many years. > The current generation is about 8 years old. The rendering is done in > MSVC on WindowsXP workstations. The type of display depends on the > processing performed, but a typical one is the LOFAR gram I mentioned. > The output of each FFT (after further massaging) cascades down the display > like a waterfall. Almost all of the of the processing is performed on > dedicated PowerPCs in a chassis separate from the workstations. The > signal processors and workstations communicate via a network.
Plenty of processing power, in other words.
> I trust the displays. I trust the rest of the processing. I don't know > about my method. If I look at the result of just my filtering in Matlab > it looks okay. But once I place this filtering at the head of the > processing chain on our system and run the same data from front to back, > the pattern shows up loud and clear in those requantized, normalized, > waterfalls. If I look at the result in another type of display that does > not do this (one that acts more like an oscilloscope and has an absolute > scale) these artifacts barely register -- they're still there, you just > wouldn't know it if you didn't know where to look.
Exactly what do the artifacts look like? Vertical stripes? Horizontal bands? With this kind of processing power available, I can't see why you want to do this clever processing. You may have a point, technically speaking, but going through all this just to save a couple of kbytes of buffers on a multi-CPU networked system seems a bit odd. I would suggest you frist make a filter that works, where you are happy about the results if not the implementation. After you have done that, you can start playing with your ideas. I suppose somebody are paying for the filters, so do make sure to meet the obligations of the contract first. If you can get your trick to work -- and justify by a cost/benefit analysis why it should be included it in the system -- you may be able to get it into the end product. Rune