Basic DSP Question

I am hoping someone can send me in the right direction for this signal
processing question, or even   supply  a simple answer. What I have is an
elementary problem in signal processing, but am unfamiliar with how to
approach the problem correctly.

I have a two part question that has been bothering me for some years, and
only recently have I been told that the answer falls within the field of
signal processing. hence I've joined the group. I am hoping you might
supply me with the two part answer; the answer may simply be to take the
mean of each segment as the best estimate, which is what I'm guessing, but
first let met state the question.

THE QUESTION:
I have a "desired signal" of fixed length I am interested in. It can be
broken into M linear component pieces. S = {S1, S2, S3, ...SM} 
Next, I have N samples of observations of this desired signal. Let us call
each of them Xi. Each can be decomposed into M linear segments as well.
An example of a desired signal might be the shape of the "average Friday"
trading day of a stock index made by  15 minute bars spaced evenly
throughout the day.
I may wish to find what the "average Friday" trading day looks like, the
desired signal, and may have 30,40, 50 etc. (N=number of observations) past
Fridays I can use to derive this chart.

My FIRST question is how to derive the signal S that looks  like the most
expected, so-to-speak, signal pattern shape. Not necessarily the average
pattern but "most likely" based on  common repetiveness.
Do I simply average all the observations for each piecewise segment, or is
it better to compute a linear combination of observations to derive the
desired signal, or is there yet another mathematical technique possible
(and what is it, such as maximum likelihood) that would produce the desired
signal? If the goal is to derive the most probable expected signal shape,
which might mean eliminating observations that look very different than
others that  are looking very similar to one another, how should I do that?
Second, what measure can I produce that would tell me how close the various
observed signals in the observation set look to one another? 

In other words, I can think of two scenarios that might answer the
question, but I don't know if either is best:
1) Take the average of all samples to produce the best estimate for
section S1 of the desired signal
S1 = ( x1,1 + x2,1 + x3,1 + ... xn,1 )/n    for segment 1
S2 = ( x1,2 + x2,2 + x3,2 + ... xn,2 )/n    for segment 2
..
SM = ( x1,m + x2,m + x3,m + ... xn,m )/n    for segment m

2) Take a linear combination of the samples to produce the best estimate
for the sections S1...SM of the signal. Coefficients a,b,c  are the same
for all equations.
S1 = a* x1,1 + b* x2,1 + c* x3,1 + ...  k* xn,1
S2 = a* x1,2 + b* x2,2 + c* x3,2 + ...  k* xn,2
..
SM = a* x1,m + b*x2,m + c* x3,m + ... k* xn,m

Is there another method that would provide a better estimate of the signal
shape S that would get rid of outlier observations that really do not look
like all the others? What methodology would derive the most probable
looking  pattern for this Friday, for instance, of stock market trading
given independent observations of previous Fridays or observations of other
data sets expected to be most like Friday's signal pattern?

QUESTION #2
The second question is the following: what measure can I produce that
would tell me how close the observed signals look like one another? In
other words, is there a measure of fitness I can use if I take some
mathematical formula to compute a desired signal, and that fitness measure
(such as a sum of squared errors) would tell me how close the observations
look like one another? 

I can restate the question another way. Given various subsets of
observations, is there some measure I can compute for each group that tells
me the individuals in one set look more like each other than the
individuals in another set?  That way I can tell if one set is superior in
predicting the signal shape S since most of its members look more closely
like each other than any other set I can come up with.

For instance, let us take the  intraday shape of trading for a stock index
once again to make matters clear. Let's assume that I have various
independent groups of data that I believe may look like this coming
Friday's shape, which is what I want to predict. One group of estimates of
the day's trading pattern might be 30 Fridays. If this Friday is a
pre-holiday, perhaps I have 20 other pre-holidays  I can put in a group and
use the answer from #1 to derive an expected signal. Or perhaps Friday is a
Federal reserve announcement day, and I have 40 such instances that can be
put in a group. Or perhaps I have 60 days where the trading bar pattern
shape on Thursday did something special, and since that's what happened so
I expect a similar follow-up of that set for Friday. 

Let's say I have multiple different subsets of days I can use to
guess/estimate what tomorrow's trading day will look like, the desired
signal S. If in looking at these various groups I find one where the group
members looked more alike each other than for other groups, that might be
the desired signal prediction for the day in question. It would then be a
question of trying different classification schemes to see which ones came
up with closest fits of member shapes to one another, so-to-speak. My
question is, what one or two mathematical measures can I compute that will
tell me how close the members within a set most looked like one another
(which might mean how closely the derived signal curve best fit the
observations)? Is that possible?

I realize this is not an typical DSP question, and perhaps is too basic,
but it has bothered me for years searching for a way to even describe the
question, and only recently did I find that this falls within the field of
signal processing. Would someone be so kind enough as to provide a possible
answer.

wbodri wrote:
> I am hoping someone can send me in the right direction for this signal
> processing question, or even   supply  a simple answer. What I have is an
> elementary problem in signal processing, but am unfamiliar with how to
> approach the problem correctly.
> 
> I have a two part question that has been bothering me for some years, and
> only recently have I been told that the answer falls within the field of
> signal processing. hence I've joined the group. I am hoping you might
> supply me with the two part answer; the answer may simply be to take the
> mean of each segment as the best estimate, which is what I'm guessing, but
> first let met state the question.
> 
> THE QUESTION:
> I have a "desired signal" of fixed length I am interested in. It can be
> broken into M linear component pieces. S = {S1, S2, S3, ...SM} 
> Next, I have N samples of observations of this desired signal. Let us call
> each of them Xi. Each can be decomposed into M linear segments as well.
> An example of a desired signal might be the shape of the "average Friday"
> trading day of a stock index made by  15 minute bars spaced evenly
> throughout the day.
> I may wish to find what the "average Friday" trading day looks like, the
> desired signal, and may have 30,40, 50 etc. (N=number of observations) past
> Fridays I can use to derive this chart.
> 
> My FIRST question is how to derive the signal S that looks  like the most
> expected, so-to-speak, signal pattern shape. Not necessarily the average
> pattern but "most likely" based on  common repetiveness.
> Do I simply average all the observations for each piecewise segment, or is
> it better to compute a linear combination of observations to derive the
> desired signal, or is there yet another mathematical technique possible
> (and what is it, such as maximum likelihood) that would produce the desired
> signal? If the goal is to derive the most probable expected signal shape,
> which might mean eliminating observations that look very different than
> others that  are looking very similar to one another, how should I do that?
> Second, what measure can I produce that would tell me how close the various
> observed signals in the observation set look to one another? 
> 
> In other words, I can think of two scenarios that might answer the
> question, but I don't know if either is best:
> 1) Take the average of all samples to produce the best estimate for
> section S1 of the desired signal
> S1 = ( x1,1 + x2,1 + x3,1 + ... xn,1 )/n    for segment 1
> S2 = ( x1,2 + x2,2 + x3,2 + ... xn,2 )/n    for segment 2
> ..
> SM = ( x1,m + x2,m + x3,m + ... xn,m )/n    for segment m
> 
> 2) Take a linear combination of the samples to produce the best estimate
> for the sections S1...SM of the signal. Coefficients a,b,c  are the same
> for all equations.
> S1 = a* x1,1 + b* x2,1 + c* x3,1 + ...  k* xn,1
> S2 = a* x1,2 + b* x2,2 + c* x3,2 + ...  k* xn,2
> ..
> SM = a* x1,m + b*x2,m + c* x3,m + ... k* xn,m
> 
> Is there another method that would provide a better estimate of the signal
> shape S that would get rid of outlier observations that really do not look
> like all the others? What methodology would derive the most probable
> looking  pattern for this Friday, for instance, of stock market trading
> given independent observations of previous Fridays or observations of other
> data sets expected to be most like Friday's signal pattern?
> 
> QUESTION #2
> The second question is the following: what measure can I produce that
> would tell me how close the observed signals look like one another? In
> other words, is there a measure of fitness I can use if I take some
> mathematical formula to compute a desired signal, and that fitness measure
> (such as a sum of squared errors) would tell me how close the observations
> look like one another? 
> 
> I can restate the question another way. Given various subsets of
> observations, is there some measure I can compute for each group that tells
> me the individuals in one set look more like each other than the
> individuals in another set?  That way I can tell if one set is superior in
> predicting the signal shape S since most of its members look more closely
> like each other than any other set I can come up with.
> 
> For instance, let us take the  intraday shape of trading for a stock index
> once again to make matters clear. Let's assume that I have various
> independent groups of data that I believe may look like this coming
> Friday's shape, which is what I want to predict. One group of estimates of
> the day's trading pattern might be 30 Fridays. If this Friday is a
> pre-holiday, perhaps I have 20 other pre-holidays  I can put in a group and
> use the answer from #1 to derive an expected signal. Or perhaps Friday is a
> Federal reserve announcement day, and I have 40 such instances that can be
> put in a group. Or perhaps I have 60 days where the trading bar pattern
> shape on Thursday did something special, and since that's what happened so
> I expect a similar follow-up of that set for Friday. 
> 
> Let's say I have multiple different subsets of days I can use to
> guess/estimate what tomorrow's trading day will look like, the desired
> signal S. If in looking at these various groups I find one where the group
> members looked more alike each other than for other groups, that might be
> the desired signal prediction for the day in question. It would then be a
> question of trying different classification schemes to see which ones came
> up with closest fits of member shapes to one another, so-to-speak. My
> question is, what one or two mathematical measures can I compute that will
> tell me how close the members within a set most looked like one another
> (which might mean how closely the derived signal curve best fit the
> observations)? Is that possible?
> 
> I realize this is not an typical DSP question, and perhaps is too basic,
> but it has bothered me for years searching for a way to even describe the
> question, and only recently did I find that this falls within the field of
> signal processing. Would someone be so kind enough as to provide a possible
> answer.
> 
Question #1:

Yes, there are various ways that you could take a number of recorded 
days of trading and try to find a 'typical' or 'average' day.  Some sort 
  of least-squares best fit on a basis set of functions may be your best 
bet (i.e. the Fourier transform uses a basis set of sinusoids -- you may 
find that a transform using a different basis set may work better).

Question #2:

If you treat question #1 as an optimization problem, where you're trying 
to find the best set of parameters to weigh basis functions, or 
otherwise optimizing an estimate, then just use the cost function from 
your optimization process.

Question #0:

No, you didn't ask it, but you should have.

If you're _really_ trying to predict stock prices keep in mind that it's 
not a field where past behavior is a good predictor of future behavior. 
  The stock market does what the stock market does, and the whiz kids 
are all kids because after they write their books on how to win big on 
the stock market it inevitably does something unpredictable.  You may be 
able to figure out what _was_ a typical day in the past, you may even be 
able to figure out what _might_ be a typical day next Friday, but the 
next big financial snafu (or volcanic eruption, or hurricane) is going 
to blow your 'quality of fit' measure into the toilet.

Question #3:

You didn't ask this one either.

If you're doing this as an academic exercise and someone else is paying 
for it -- go for it!

Be careful with staking too much of your own money and reputation on 
being able to predict the stock market, though.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" gives you just what it says.
See details at http://www.wescottdesign.com/actfes/actfes.html

Tim Wescott <tim@seemywebsite.com> writes:
> [...several stock market analysis suggestions snipped...]

Tim,

Have you ever studied Ito calculus? If so, is it just as unsuccessful
at handling stock market predictions as any other tool?
-- 
%  Randy Yates                  % "Maybe one day I'll feel her cold embrace,
%% Fuquay-Varina, NC            %                    and kiss her interface, 
%%% 919-577-9882                %            til then, I'll leave her alone."
%%%% <yates@ieee.org>           %        'Yours Truly, 2095', *Time*, ELO   
http://www.digitalsignallabs.com

On 19 Mai, 12:49, Randy Yates <ya...@ieee.org> wrote:
> Tim Wescott <t...@seemywebsite.com> writes:
> > [...several stock market analysis suggestions snipped...]
>
> Tim,
>
> Have you ever studied Ito calculus? If so, is it just as unsuccessful
> at handling stock market predictions as any other tool?

Yesterday I saw a newspaper article stating that oil prices
are expected to average $140/barrel in the second half of
2008. The reason for this rise in expected prices (and no, I don't
know what the previous prediction was) is the recent
earthquakes in China over last week or so.

Would your friend Ito have been able to predict that sort
of thing to happen based on the information available two
weeks ago?

Rune

Rune Allnor <allnor@tele.ntnu.no> writes:
> [...]
> Would your friend Ito have been able to predict that sort
> of thing to happen based on the information available two
> weeks ago?

Do you follow the yellow brick road? Do you believe in
Unicorns and Centaurs? Is Harry Potter your friend?
-- 
%  Randy Yates                  % "Ticket to the moon, flight leaves here today 
%% Fuquay-Varina, NC            %  from Satellite 2"
%%% 919-577-9882                % 'Ticket To The Moon' 
%%%% <yates@ieee.org>           % *Time*, Electric Light Orchestra
http://www.digitalsignallabs.com

Randy Yates wrote:
> Rune Allnor <allnor@tele.ntnu.no> writes:
>> [...]
>> Would your friend Ito have been able to predict that sort
>> of thing to happen based on the information available two
>> weeks ago?
> 
> Do you follow the yellow brick road? Do you believe in
> Unicorns and Centaurs? Is Harry Potter your friend?

Lots of toads knew about the quake a couple of days before it happened. 
Maybe those were friends on Harry Potter. They sure seem smarter than 
humans.

Steve

Steve Underwood <steveu@dis.org> writes:

> Randy Yates wrote:
>> Rune Allnor <allnor@tele.ntnu.no> writes:
>>> [...]
>>> Would your friend Ito have been able to predict that sort
>>> of thing to happen based on the information available two
>>> weeks ago?
>>
>> Do you follow the yellow brick road? Do you believe in
>> Unicorns and Centaurs? Is Harry Potter your friend?
>
> Lots of toads knew about the quake a couple of days before it
> happened. Maybe those were friends on Harry Potter. They sure seem
> smarter than humans.

It seems I remember reading somewhere about precursory subsonic waves -
perhaps the toads are sensitive to those? 

However, I believe Harry Potter was fluent in snake.
-- 
%  Randy Yates                  % "She has an IQ of 1001, she has a jumpsuit
%% Fuquay-Varina, NC            %            on, and she's also a telephone."
%%% 919-577-9882                % 
%%%% <yates@ieee.org>           %        'Yours Truly, 2095', *Time*, ELO   
http://www.digitalsignallabs.com

On 19 Mai, 14:32, Randy Yates <ya...@ieee.org> wrote:
> Rune Allnor <all...@tele.ntnu.no> writes:
> > [...]
> > Would your friend Ito have been able to predict that sort
> > of thing to happen based on the information available two
> > weeks ago?
>
> Do you follow the yellow brick road? Do you believe in
> Unicorns and Centaurs? Is Harry Potter your friend?

My very simple point was that some arithmetics that
can not predict major stuff like earthquakes or wars,
can not possibly be used to predict stock market data,
for the very simple reason that earthquakes and wars
happen to influence the stock market.

I know it comes as a surprise to lots of people, but
sme very basic understanding of mere trivial aspects
of the 'data domain' (the processes that generate the
data in question) goes a very long way to determine
what can be done and what can not be done.

Rune

On May 19, 10:38 am, Rune Allnor <all...@tele.ntnu.no> wrote:
> On 19 Mai, 14:32, Randy Yates <ya...@ieee.org> wrote:
>
> > Rune Allnor <all...@tele.ntnu.no> writes:
> > > [...]
> > > Would your friend Ito have been able to predict that sort
> > > of thing to happen based on the information available two
> > > weeks ago?
>
> > Do you follow the yellow brick road? Do you believe in
> > Unicorns and Centaurs? Is Harry Potter your friend?
>
> My very simple point was that some arithmetics that
> can not predict major stuff like earthquakes or wars,
> can not possibly be used to predict stock market data,
> for the very simple reason that earthquakes and wars
> happen to influence the stock market.
>
> I know it comes as a surprise to lots of people, but
> sme very basic understanding of mere trivial aspects
> of the 'data domain' (the processes that generate the
> data in question) goes a very long way to determine
> what can be done and what can not be done.
>
> Rune

An aspect of stock-market analysis using sampled data that puzzles me
is aliasing. How do the pundits apply anti-alias filtering to the raw
data before they sample it?

Daily closing prices, daily river-height measurements, daily high-low
temperature measurements .. these are all informative, but they do not
suffice as fodder for DSP methods that yield daily data. What is the
crossover between statistical and DSP manipulation?

Jerry

On 19 Mai, 16:58, Jerry Avins <j...@ieee.org> wrote:
> On May 19, 10:38 am, Rune Allnor <all...@tele.ntnu.no> wrote:
>
>
>
>
>
> > On 19 Mai, 14:32, Randy Yates <ya...@ieee.org> wrote:
>
> > > Rune Allnor <all...@tele.ntnu.no> writes:
> > > > [...]
> > > > Would your friend Ito have been able to predict that sort
> > > > of thing to happen based on the information available two
> > > > weeks ago?
>
> > > Do you follow the yellow brick road? Do you believe in
> > > Unicorns and Centaurs? Is Harry Potter your friend?
>
> > My very simple point was that some arithmetics that
> > can not predict major stuff like earthquakes or wars,
> > can not possibly be used to predict stock market data,
> > for the very simple reason that earthquakes and wars
> > happen to influence the stock market.
>
> > I know it comes as a surprise to lots of people, but
> > sme very basic understanding of mere trivial aspects
> > of the 'data domain' (the processes that generate the
> > data in question) goes a very long way to determine
> > what can be done and what can not be done.
>
> > Rune
>
> An aspect of stock-market analysis using sampled data that puzzles me
> is aliasing. How do the pundits apply anti-alias filtering to the raw
> data before they sample it?

Well stock market data are discrete time, not continuous. Even if
there
are huge amounts of transactions going on, the number is finite. So
stock market data ader discrete in nature. No anti-alias filter
required.

> Daily closing prices, daily river-height measurements, daily high-low
> temperature measurements .. these are all informative, but they do not
> suffice as fodder for DSP methods that yield daily data. What is the
> crossover between statistical and DSP manipulation?

I have found that 'econometrics' seems to be an important application
for fancy data analysis methods. The book on Kalman filters (State
space
analysis of time series) use economical data for case studies.

Rune