I have a rather preliminary question about filter design.. Am I right
in saying that an implicit assumption in filter design is that the
sampling
is uniform ? Just like in FFT, where data has to be uniformly sampled.
I have data which is non-uniformly sampled, and I have to develop a
second-order HPF that can work on the data to reject low-freq.
components. Is there a specific way to proceed with this?

Thank you
Dinkar

"dnb" <d...@yahoo.com> wrote in message
news:f...@posting.google.com...
> [...]
> I have data which is non-uniformly sampled, and I have to develop a
> second-order HPF that can work on the data to reject low-freq.
> components. Is there a specific way to proceed with this?

The most practical approach would probably be to resample the data uniformly
before filtering.  If you really need to, you can regenerate the samples at
non-uniform abscissas when you're done.  This isolates the filter design
from the major decisions you need to make about how to reconstruct the
original signal from the non-uniform samples.

--
Matt

"Matt Timmermans" <m...@sympatico.nospam-remove.ca> wrote in message
news:<mpENc.13316$B...@news20.bellglobal.com>...
> "dnb" <d...@yahoo.com> wrote in message
> news:f...@posting.google.com...
> > [...]
> > I have data which is non-uniformly sampled, and I have to develop a
> > second-order HPF that can work on the data to reject low-freq.
> > components. Is there a specific way to proceed with this?
> 
> The most practical approach would probably be to resample the data uniformly
> before filtering.  If you really need to, you can regenerate the samples at
> non-uniform abscissas when you're done.  This isolates the filter design
> from the major decisions you need to make about how to reconstruct the
> original signal from the non-uniform samples.

The data input is inherently non-uniformly sampled (I dont have access to the
source). I can interpolate if that is what you mean. If so, can you
kindly mention type of interpolation you would recommed?

Thank you
Dinkar

Dinkar,

     I believe that the method you use should be dependent on what
kind of data you have. For example, linear interpolation will lead to
a case where the second derivative may have very high values or even
be undefined. Depending on how you HPF works this could be a big
problem.

    I also suspect that if you shared more of the details of your
project with the group you could get a better answer. By any chance,
is this related to the stock market?

                                       Bob Sherry


"dnb" <d...@yahoo.com> wrote in message
news:f...@posting.google.com...
> "Matt Timmermans" <m...@sympatico.nospam-remove.ca> wrote in
message news:<mpENc.13316$B...@news20.bellglobal.com>...
> > "dnb" <d...@yahoo.com> wrote in message
> > news:f...@posting.google.com...
> > > [...]
> > > I have data which is non-uniformly sampled, and I have to
develop a
> > > second-order HPF that can work on the data to reject low-freq.
> > > components. Is there a specific way to proceed with this?
> >
> > The most practical approach would probably be to resample the data
uniformly
> > before filtering.  If you really need to, you can regenerate the
samples at
> > non-uniform abscissas when you're done.  This isolates the filter
design
> > from the major decisions you need to make about how to reconstruct
the
> > original signal from the non-uniform samples.
>
> The data input is inherently non-uniformly sampled (I dont have
access to the
> source). I can interpolate if that is what you mean. If so, can you
> kindly mention type of interpolation you would recommed?
>
> Thank you
> Dinkar

"dnb" <d...@yahoo.com> wrote in message
news:f...@posting.google.com...
> The data input is inherently non-uniformly sampled (I dont have access to
the
> source). I can interpolate if that is what you mean.

Yes.

> If so, can you
> kindly mention type of interpolation you would recommed?

Ah, I see.  I was leaving that decision to you, because it depends entirely
on your application.  When we use uniform sampling in DSP, we almost always
ensure or assume that anti-aliasing filters have been applied to the signal
before sampling, and that the sample rate has been chosen according the
Nyquist criterion.  These conditions imply the correct way to reconstruct
the signal (i.e., interpolate) mathematically, tells us how the samples
relate to the signal in the real world, and gives physical meaning to
processes like high-pass filtering.

There are no such common assumptions that usually apply to applications of
non-uniform sampling, so you will have to tell us about the process that
generates the samples before we could determine how interpolation would best
be accomplished.

"Matt Timmermans" <m...@sympatico.nospam-remove.ca> wrote in message
news:<CSZNc.675$U...@news20.bellglobal.com>...
> "dnb" <d...@yahoo.com> wrote in message
> news:f...@posting.google.com...
> > The data input is inherently non-uniformly sampled (I dont have access to
>  the
> > source). I can interpolate if that is what you mean.
> 
> Yes.
> 
> > If so, can you
> > kindly mention type of interpolation you would recommed?
> 
> Ah, I see.  I was leaving that decision to you, because it depends entirely
> on your application.  When we use uniform sampling in DSP, we almost always
> ensure or assume that anti-aliasing filters have been applied to the signal
> before sampling, and that the sample rate has been chosen according the
> Nyquist criterion.  These conditions imply the correct way to reconstruct
> the signal (i.e., interpolate) mathematically, tells us how the samples
> relate to the signal in the real world, and gives physical meaning to
> processes like high-pass filtering.
> 
> There are no such common assumptions that usually apply to applications of
> non-uniform sampling, so you will have to tell us about the process that
> generates the samples before we could determine how interpolation would best
> be accomplished.


Bob and Matt,

Thanks much for the reply. Bob - this is not a project for Wall
Street, and I do not work in the financial world! I work in the
digital video domain. Clock samples are sent in the stream by an
encoder, but they are non-uniformly spaced. The samples may deviate
from an ideal clock,  and I am trying to measure how far the samples
deviate from the ideal clock. Only the high-frequency components in
the deviation from the ideal clock are a problem (hence the HPF). One
constraint is that successive samples of the clock do not exceed a
certain time (100ms). In other words, I can have samples as:
a(t=0), a(tPms), a(tums), a(t™ms), a(t0ms). The next sample
cannot be
a(t"5)ms. One other fact is one can find a good linear fit between
the samples.

Dinkar.

Hi Dinkar,

If you really want to do filtering, then you probably won't go too far wrong
by using a simple cubic spline interpolation, resampling at a higher,
uniform, rate, and filtering normally.  (I would normally find you a good
reference for the type of interpolation I mean, but I'm on dial-up access
this week, and that makes the internet no fun).

I don't know that the conclusions you would draw from the resulting data
would be terribly useful, however.  The trouble is that the nature of the
highest frequency components in the interpolated signal will depend a lot on
the method of interpolation, and there doesn't seem to be anything known
about this non-ideal clock that would give you physical justification for
choosing one particular method, so there's a good chance that you'll be
analyzing information you invented instead of what's really in the signal.

You would probably be better off using a low-pass filter on the interpolated
data, with a cutoff frequency below 5Hz (derived from your 100ms
constraint), and then analyzing the differences between each sample and the
low-passed signal.  The low-pass filter will remove most of the artifacts
introduced by the choice of interpolation method.

If your final application is trying to estimating buffering requirements
based on packet jitter, or something like that, the above technique matches
the application -- there is some part of the system with a low-pass response
to the clock, leaving you with a requirement to handle the high-frequency
residual.  If that is the case, then you want the low-pass filter above to
match the one in the system under test.

If I'm way off about what you're trying to do, then you probably want to
google for "jitter analysis".

--

Matt

In article <f...@posting.google.com>,
dnb <d...@yahoo.com> wrote:
>Thanks much for the reply. Bob - this is not a project for Wall
>Street, and I do not work in the financial world!

I've been lurking here for a while, and even posted a few times.  I
am curious about the statement above, because I have seen similar
sentiments expressed in other posts.  What is it about DSP applied
to financial analysis that seems to turn folks off here?  Me, my
background is physics and I work in the defense industry, but I do
find it interesting to invent filters and apply them to market data
problems; in fact much of what I learned about digital filters for
tracking targets in a military context, came from looking at such
filters applied to market movements.

So what's the problem?  Shed some light please.

-A

a...@spamcop.net (axlq) wrote in message news:<cecpt1$f99$1...@blue.rahul.net>...
> In article <f...@posting.google.com>,
> dnb <d...@yahoo.com> wrote:
> >Thanks much for the reply. Bob - this is not a project for Wall
> >Street, and I do not work in the financial world!
> 
> I've been lurking here for a while, and even posted a few times.  I
> am curious about the statement above, because I have seen similar
> sentiments expressed in other posts.  What is it about DSP applied
> to financial analysis that seems to turn folks off here?  Me, my
> background is physics and I work in the defense industry, but I do
> find it interesting to invent filters and apply them to market data
> problems; in fact much of what I learned about digital filters for
> tracking targets in a military context, came from looking at such
> filters applied to market movements.
> 
> So what's the problem?  Shed some light please.
> 
> -A

The problem, as I see it, is that financial data are completely random, 
while "classical" DSP problems (i.e. applications in communications or 
physics) have to obey some underlying system. One knows, for instance,
that a communications link will have certain statistical properties and 
that these properties will be more or less stable when the link is in use. 
One knows that certain targets will stay in a combat theatre, and move 
smoothly under the laws of physics, until they leave or are destroyed.
A 688 class sub will move differently than an F22 raptor, but both move
within their respective physical limits which are constant. 

Compare this to, say, a stock market heavily influenced by Enron shares. 
The market index is suceptible to manipulation, and random crashes. There 
is no way of knowing whether a stock price or a market index is manipulated
or not. There is no way of predicting news bulletins a la Enron, that 
completely changes the situation. There is no way of predicting and 
accounting for some political decision to initiate or remove various tax 
schemes.

The main difference between "classical" DSP and finacial data is that 
it is virtually impossible to use past financial data in the design of 
future DSP systems. DSP terminoøogy and concepts can be used to describe
past data. I don't see how DSP tools can be useful for making predictions 
on financial data.

Rune

axlq wrote:

> In article <f...@posting.google.com>,
> dnb <d...@yahoo.com> wrote:
> 
>>Thanks much for the reply. Bob - this is not a project for Wall
>>Street, and I do not work in the financial world!
> 
> 
> I've been lurking here for a while, and even posted a few times.  I
> am curious about the statement above, because I have seen similar
> sentiments expressed in other posts.  What is it about DSP applied
> to financial analysis that seems to turn folks off here?  Me, my
> background is physics and I work in the defense industry, but I do
> find it interesting to invent filters and apply them to market data
> problems; in fact much of what I learned about digital filters for
> tracking targets in a military context, came from looking at such
> filters applied to market movements.
> 
> So what's the problem?  Shed some light please.
> 
> -A

I happen to think that the fate of companies and the worth of their
stock depends more on how they are managed and other real-world events
far more than past history of ups and downs in price. "Technical
analysis" tries to predict future prices by finding trends in past
prices, with little or no attention to "outside" variables. I think it's
a crock.

That's not why I suspect DSP analyses of stock prices. The techniques of
DSP analysis apply to bandlimited data. In order for a continuous signal
to be validly sampled, it must be established that the signal contains
no frequencies as high as half the sample rate. In some cases, that is
known a priori by the nature of the signal. At all other cases, the
continuous signal must be filtered to remove high frequencies before
sampling takes place. In the market analyses I've seen, no such
filtering is done. Indeed, the way the data are collected, it couldn't
be. No allowance for this lapse or estimate of its effects has ever come
to my attention. The Nyquist criterion is routinely violated. Data
collected once a day are used to predict daily trends.

Jerry
-- 
Engineering is the art of making what you want from things you can get.
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯