Is this 'clever' method of filtering legit?

I'm getting strange results from 'clever' filtering -- is what I'm doing
legit?  There's often a moire-like pattern in the output, even though the
filtering tamps down the aliasing as intended.

My source is digitized acoustic data replayed from wave files.  The replay
rate can be 1x, 2x, 4x or 8x.  For rates > 1x I filter the data (and
desample) to knock down frequencies that would alias into the new passband
(1/2, 1/4 or 1/8th the original).

I receive the data in blocks that are multiples of 1k.  What I've done is
apply two different length filters to each block as I receive it.  The
length of both filters divides evenly into the size of the data block. 
The first is a long filter (in relative terms) and the second is equal in
length to the decimation factor (i.e., the replay rate).  I apply the long
filter to the input data, save the result, shift the data by the decimation
factor and repeat.  I do this as long as the tail of the long filter does
not go beyond the block of input data.  Then, from the place where the
next dot product would be formed and from then on I apply the short filter
in the same manner until it too would otherwise go beyond the block of
input data.

For example, if I had a replay rate of 2x and had 32 input points and my
long filter was of length 8, and my short filter of length 2, I could
apply the long filter 13 times (shifting the data by two after each
application) then apply the short filter 3 times (shifting the data by two
after each application).  This would give me 16 output points as desired.

After further processing I display the output in a LOFAR gram (a waterfall
of frequency (horizontal) vs. time (vertical)) and I see this striping...
like blinds turned on edge.  Is this because there's a flaw in my 'clever'
method or do you think there is some other source for this?

My goal in this is simultaneously to a) avoid the need to incorporate
buffering in the existing software and b) get better results than I would
if I repeatedly applied a filter of length equal to the decimation
factor.

Thanks guys.

Martin

MartinC skrev:

> After further processing I display the output in a LOFAR gram (a waterfall
> of frequency (horizontal) vs. time (vertical)) and I see this striping...
> like blinds turned on edge.  Is this because there's a flaw in my 'clever'
> method or do you think there is some other source for this?

Start with the easy stuff first: What type of data do you process?
How do you plot the results? Have you tested the plotting routine
with other data? 

Rune

>
>MartinC skrev:
>
>> After further processing I display the output in a LOFAR gram (a
waterfall
>> of frequency (horizontal) vs. time (vertical)) and I see this
striping...
>> like blinds turned on edge.  Is this because there's a flaw in my
'clever'
>> method or do you think there is some other source for this?
>
>Start with the easy stuff first: What type of data do you process?
>How do you plot the results? Have you tested the plotting routine
>with other data? 
>
>Rune
>
>

Hi Rune,

The system accepts data from multiple sources, both analog and digital. 
My starting point was to look at the case of 32 bit wave files so that I
could more easily control the inputs and make them repeatable, get that
working, then integrate the code into various places in the system.

The system has been around in one form or another for many, many years. 
The current generation is about 8 years old.  The rendering is done in
MSVC on WindowsXP workstations.  The type of display depends on the
processing performed, but a typical one is the LOFAR gram I mentioned. 
The output of each FFT (after further massaging) cascades down the display
like a waterfall.  Almost all of the of the processing is performed on
dedicated PowerPCs in a chassis separate from the workstations.  The
signal processors and workstations communicate via a network.

I trust the displays.  I trust the rest of the processing.  I don't know
about my method.  If I look at the result of just my filtering in Matlab
it looks okay.  But once I place this filtering at the head of the
processing chain on our system and run the same data from front to back,
the pattern shows up loud and clear in those requantized, normalized,
waterfalls.  If I look at the result in another type of display that does
not do this (one that acts more like an oscilloscope and has an absolute
scale) these artifacts barely register -- they're still there, you just
wouldn't know it if you didn't know where to look.

Thanks,

Martin

MartinC skrev:
> >
> >MartinC skrev:
> >
> >> After further processing I display the output in a LOFAR gram (a
> waterfall
> >> of frequency (horizontal) vs. time (vertical)) and I see this
> striping...
> >> like blinds turned on edge.  Is this because there's a flaw in my
> 'clever'
> >> method or do you think there is some other source for this?
> >
> >Start with the easy stuff first: What type of data do you process?
> >How do you plot the results? Have you tested the plotting routine
> >with other data?
> >
> >Rune
> >
> >
>
> Hi Rune,
>
> The system accepts data from multiple sources, both analog and digital.
> My starting point was to look at the case of 32 bit wave files so that I
> could more easily control the inputs and make them repeatable, get that
> working, then integrate the code into various places in the system.
>
> The system has been around in one form or another for many, many years.
> The current generation is about 8 years old.  The rendering is done in
> MSVC on WindowsXP workstations.  The type of display depends on the
> processing performed, but a typical one is the LOFAR gram I mentioned.
> The output of each FFT (after further massaging) cascades down the display
> like a waterfall.  Almost all of the of the processing is performed on
> dedicated PowerPCs in a chassis separate from the workstations.  The
> signal processors and workstations communicate via a network.

Plenty of processing power, in other words.

> I trust the displays.  I trust the rest of the processing.  I don't know
> about my method.  If I look at the result of just my filtering in Matlab
> it looks okay.  But once I place this filtering at the head of the
> processing chain on our system and run the same data from front to back,
> the pattern shows up loud and clear in those requantized, normalized,
> waterfalls.  If I look at the result in another type of display that does
> not do this (one that acts more like an oscilloscope and has an absolute
> scale) these artifacts barely register -- they're still there, you just
> wouldn't know it if you didn't know where to look.

Exactly what do the artifacts look like? Vertical stripes?
Horizontal bands?

With this kind of processing power available, I can't see why
you want to do this clever processing. You may have a point,
technically speaking, but going through all this just to save
a couple of kbytes of buffers on a multi-CPU networked system
seems a bit odd.

I would suggest you frist make a filter that works, where you
are happy about the results if not the implementation. After
you have done that, you can start playing with your ideas.

I suppose somebody are paying for the filters, so do make
sure to meet the obligations of the contract first. If you can
get your trick to work -- and justify by a cost/benefit analysis
why it should be included it in the system -- you may be
able to get it into the end product.  

Rune

MartinC skrev:
>  I apply the long
> filter to the input data, save the result, shift the data by the decimation
> factor and repeat.  I do this as long as the tail of the long filter does
> not go beyond the block of input data.  Then, from the place where the
> next dot product would be formed and from then on I apply the short filter
> in the same manner until it too would otherwise go beyond the block of
> input data.
...
> After further processing I display the output in a LOFAR gram (a waterfall
> of frequency (horizontal) vs. time (vertical)) and I see this striping...
> like blinds turned on edge.

I knew I had missed something in this post. Your comment about the
"blindfolds on edge" made me think about vertical stripes, but maybe
you are talking about horizontal bands.

If the latter, I suspect you see the effects of applying two different
filters to the sequence; or possibly imperfections at the joints
between consecutive buffers. 

Rune

>
>I knew I had missed something in this post. Your comment about the
>"blindfolds on edge" made me think about vertical stripes, but maybe
>you are talking about horizontal bands.
>
>If the latter, I suspect you see the effects of applying two different
>filters to the sequence; or possibly imperfections at the joints
>between consecutive buffers. 
>
>Rune
>
>

You were correct the first time, Rune.  The problem is very narrow
vertical banding.  The spacing is on the order of 1.6, 3.2, 6.4 Hz (don't
hold me to those values exactly, I'd need to go back to the lab to do more
checking) and at first blush those numbers appear to track the replay rate.
 The entire frequency range we capture is variable, but ~2560 Hz
(basebanded) would be the limit at a replay rate of 8.

As to your point about "why not just go ahead and buffer", well, given my
druthers, I would do that.

Martin

MartinC skrev:
> >
> >I knew I had missed something in this post. Your comment about the
> >"blindfolds on edge" made me think about vertical stripes, but maybe
> >you are talking about horizontal bands.
> >
> >If the latter, I suspect you see the effects of applying two different
> >filters to the sequence; or possibly imperfections at the joints
> >between consecutive buffers.
> >
> >Rune
> >
> >
>
> You were correct the first time, Rune.  The problem is very narrow
> vertical banding.  The spacing is on the order of 1.6, 3.2, 6.4 Hz (don't
> hold me to those values exactly, I'd need to go back to the lab to do more
> checking) and at first blush those numbers appear to track the replay rate.
>  The entire frequency range we capture is variable, but ~2560 Hz
> (basebanded) would be the limit at a replay rate of 8.

OK. Seems you would be looking for zeros in the filter transfer
functions, then. Or could there be some pre-plotter spectrum
interpolation that is missing?

Rune

MartinC skrev:
> >
> >I knew I had missed something in this post. Your comment about the
> >"blindfolds on edge" made me think about vertical stripes, but maybe
> >you are talking about horizontal bands.
> >
> >If the latter, I suspect you see the effects of applying two different
> >filters to the sequence; or possibly imperfections at the joints
> >between consecutive buffers.
> >
> >Rune
> >
> >
>
> You were correct the first time, Rune.  The problem is very narrow
> vertical banding.  The spacing is on the order of 1.6, 3.2, 6.4 Hz (don't
> hold me to those values exactly, I'd need to go back to the lab to do more
> checking) and at first blush those numbers appear to track the replay rate.

OK. Sounds like you should be looking for zeros in the filter
transfer function, or poor pre-plotting interpolation. In a different
post you said you trust the plotting software. Could there be
some spectrum interpolation step between the filter and the
plotter in the previous versions of the SW, that you have not
included? 

Rune

MartinC skrev:
> >
> >I knew I had missed something in this post. Your comment about the
> >"blindfolds on edge" made me think about vertical stripes, but maybe
> >you are talking about horizontal bands.
> >
> >If the latter, I suspect you see the effects of applying two different
> >filters to the sequence; or possibly imperfections at the joints
> >between consecutive buffers.
> >
> >Rune
> >
> >
>
> You were correct the first time, Rune.  The problem is very narrow
> vertical banding.  The spacing is on the order of 1.6, 3.2, 6.4 Hz (don't
> hold me to those values exactly, I'd need to go back to the lab to do more
> checking) and at first blush those numbers appear to track the replay rate.

OK. Sounds like you should be looking for zeros in the filter
transfer function, or poor pre-plotting interpolation. In a different
post you said you trust the plotting software. Could there be
some spectrum interpolation step between the filter and the
plotter in the previous versions of the SW, that you have not
included? 

Rune

Hi Martin,

I think that your change in filter coefficient sets results in a time
varying change in frequency response that results in a modulation of
the data.

The MATLAB 'filter' function used with A = 1.0 for a FIR filter
implementation and using the Zf and Zi parameters is an example of the
minimum size of storage to just read the data in once. try 'help
filter' in Matlab if you have it.

If you can't afford to store this much data per channel you can avoid
storage by reading in a block of data of (blocksize + filterlength - 1)
size every blocksize samples and just running the normal length filter
to produce the correct number of outputs per block. This trades off a
little data bandwidth for maintaining data storage between filter
operations.

Dale Dalrymple
http://dbdimages.com