axlq@spamcop.net (axlq) wrote in message news:<cee658$9ol$1@blue.rahul.net>...
> In article <f56893ae.0407300324.1fcb45fc@posting.google.com>,
> Rune Allnor <allnor@tele.ntnu.no> wrote:
> >axlq@spamcop.net (axlq) wrote in message news:<cecpt1$f99$1@blue.rahul.net>...
> >> So what's the problem? Shed some light please.
> >
> >The problem, as I see it, is that financial data are completely random,
> >while "classical" DSP problems (i.e. applications in communications or
> >physics) have to obey some underlying system.
>
> I guess that would be a problem, if financial data were completely
> random. Completely random financial data would have the same
> statistical character as a random walk -- i.e. the distribution
> of returns are gaussian.
>
> The fact that financial data *doesn't* exhibit this gaussian
> characteristic implies that some determinism is taking place, an
> idea embodied by the Fractal Market Hypothesis (FMH) contradicting
> the traditional Efficient Market Hypothesis (EMH) which posits
> random walks. The fact that the distributions aren't gaussian
> (more like "stable paretian" having a variance of infinity) renders
> worthless all the usual statistical analysis techniques one would
> apply (standard deviation and the like) but that's another subject
> entirely. Anyway, the distribution exhibited by markets implies
> that market trends will last longer than with a random walk, and
> that a market will be prone to large dramatic moves more often.
>
> That said, DSP techniques applied *properly* to financial data --
> say, to remove noise from long-term trends that do exist -- can
> be useful. The successful commodity traders I have encountered
> don't waste their time trying to make predictions; instead they
> *anticipate* certain behaviors based on past history. DSP simply
> provides some tools with which to build a strategy; DSP isn't *the*
> think these traders use to make decisions.
>
> >One knows, for instance, that a communications link will have
> >certain statistical properties and that these properties will
> >be more or less stable when the link is in use. One knows that
> >certain targets will stay in a combat theatre, and move smoothly
> >under the laws of physics, until they leave or are destroyed. A
> >688 class sub will move differently than an F22 raptor, but both
> >move within their respective physical limits which are constant.
>
> This is where things get interesting, and my field of employment
> (defense) potentially overlaps the financial analysis field.
>
> There are cases where a target is unknown, the dynamics of the
> target cannot be modeled or assumed in advance, and the target is
> capable of deliberate erratic movements with accelerations so great
> that they resemble elastic collisions, AND the sensor tracking the
> target has to contend with positioning errors due to atmospheric
> disturbances and other sources of noise.
>
> This is exactly the problem faced by designers of filters for market
> data, only the financial data is a 1-dimensional time series whereas
> the tracking problem has 2 or 3 dimensions. In fiancial terms, the
> problem could be stated as "how do I track the true long-term price
> movements of this instrument amid all the short-term variations
> (noise), smoothing the trends yet maintaining the high-frequency
> component of the signal at turning points?" The target tracking
> problem is basically the same: "How do I accurately track the motion
> of a target that has occiasional high-frequency components in its
> motion, while filtering out the high-frequency positioning errors?"
>
> Mark Jurik's JMA filter is probably the best thing I've seen in
> the "financial DSP" arena, but it's a proprietary algorithm. In
> my spare time I'm coming up with ways to reverse-engineer it. I
> do know that JMA isn't a spectral filter; it's more distribution
> based, maintaining a measurement of filter error relative to a
> measurement of signal noise and somehow adapting the smoothing
> power accordingly. The result is something that can follow step
> functions with almost no overshoot and yet maintain attenuation of
> the high frequencies that are really present within the signal.
>
> See the noisy sawtooth signal at the bottom of
> http://www.jurikres.com/down/why_jma.pdf and also the noisy
> sawtooth and noisy step function response at the bottom of
> http://www.jurikres.com/down/ma_evolv.pdf -- Pretty cool, if you ask
> me.
>
> -A
You may want to look at Interacting Multiple Models (IMM). Essentially you can
look at the innovations and decide when to modify the model.
This corresponds to tracking targets that tend to move in straight lines
(constant velocity) and then perform some maneuvre.
Cheers,
David
Reply by Jerry Avins●July 31, 20042004-07-31
Rune Allnor wrote:
...
>>>The basic difference between a good and a bad trader would be that,
>>>given the same information about the market, the good trader understands
>>>the psychology of the market, and understands how other actors will
>>>react every time he recieves new pieces of information. A DSP chip can't
>>>replicate that.
>>
>>Actually, I don't see why not.
>
>
> Eh... well, I said "good trader", not "successful trader". The successful
> trader would most likely be the one that gets more information earlier
> than others. With this distinction, the DSP might very well replicate the
> success of the merely good trader...
If the market typically reacts a certain way to certain information,
even a neural net could learn to anticipate the reaction. I don't see
that knowing the motivation for the market's behavior is especially
relevant.
Jerry
--
Engineering is the art of making what you want from things you can get.
�����������������������������������������������������������������������
Reply by Rune Allnor●July 31, 20042004-07-31
Jerry Avins <jya@ieee.org> wrote in message news:<410bb04d$0$2826$61fed72c@news.rcn.com>...
> Rune Allnor wrote:
>
> >> ... DSP techniques applied *properly* to financial data --
> >>say, to remove noise from long-term trends that do exist -- can
> >>be useful.
> >
> >
> > I see acontradiction here. A few lines up you say that standard statistical
> > tools don't apply to financial analysis. Here you say that DSP techniques
> > can be used for financial analysis. To me, DSP techniques are nothing
> > more than specially tailored statistical techniques that rely on standard
> > statistical analysis (mean, standard deviation, variance) and that in
> > 95% of all cases are based on assumptions about the system being Gaussian.
> >
> > I think you might have answered your own question here.
>
> It's not an answer if he doesn't want it to be.
That would depend entirely on what "hat one wears". An engineer would
hopefully accept failing prerequisits as the reason for why a given
method doesn't work. An economist might not. And all too often does not.
> >>The successful commodity traders I have encountered
> >>don't waste their time trying to make predictions; instead they
> >>*anticipate* certain behaviors based on past history.
> >
> >
> > What would be the difference between "predict" and "anticipate"?
>
> A prediction is a statement about the future. Anticipation is action in
> advance of an event. A prediction can be shown to have been wrong. An
> action taken in anticipation of an event can be ascribed to whatever
> motive seems convenient and plausable after the fact. If "predict" and
> "anticipate" are not merely distinctions without a difference, the
> difference must be that anticipating is a bit less likely to lead to
> embarrassment.
So "It will come snow next winter" is a prediction while "I wouldn't be
surprised if snow falls next winter" is an anticipation? I could go along
with that anticipation is a slightly less certain expression of clairvoyance
than right-out prediction.
> > The basic difference between a good and a bad trader would be that,
> > given the same information about the market, the good trader understands
> > the psychology of the market, and understands how other actors will
> > react every time he recieves new pieces of information. A DSP chip can't
> > replicate that.
>
> Actually, I don't see why not.
Eh... well, I said "good trader", not "successful trader". The successful
trader would most likely be the one that gets more information earlier
than others. With this distinction, the DSP might very well replicate the
success of the merely good trader...
> Jerry
Rune
Reply by Jerry Avins●July 31, 20042004-07-31
axlq wrote:
> In article <410ac405$0$2815$61fed72c@news.rcn.com>,
> Jerry Avins <jya@ieee.org> wrote:
>
>>dnb is probably aware that Bob has a lively interest in the stock market
>>-- he is the moderator of an informal investment group on the web -- and
>>I assumed his comment to be of the "in case you're wondering" kind.
>
>
> Ah. Okay. Chalk it up to me not being familiar with who's who around
> here....
>
> -A
I imagine that very few people here knew that. It's a fine example of
how assumptions that seem eninently reasonable can turn out to be
entirely wrong. Life is too short to let us check every assumption, so
we do that sort of thing often.
Jerry
--
Engineering is the art of making what you want from things you can get.
�����������������������������������������������������������������������
Reply by Jerry Avins●July 31, 20042004-07-31
Rune Allnor wrote:
>> ... DSP techniques applied *properly* to financial data --
>>say, to remove noise from long-term trends that do exist -- can
>>be useful.
>
>
> I see acontradiction here. A few lines up you say that standard statistical
> tools don't apply to financial analysis. Here you say that DSP techniques
> can be used for financial analysis. To me, DSP techniques are nothing
> more than specially tailored statistical techniques that rely on standard
> statistical analysis (mean, standard deviation, variance) and that in
> 95% of all cases are based on assumptions about the system being Gaussian.
>
> I think you might have answered your own question here.
It's not an answer if he doesn't want it to be.
>>The successful commodity traders I have encountered
>>don't waste their time trying to make predictions; instead they
>>*anticipate* certain behaviors based on past history.
>
>
> What would be the difference between "predict" and "anticipate"?
A prediction is a statement about the future. Anticipation is action in
advance of an event. A prediction can be shown to have been wrong. An
action taken in anticipation of an event can be ascribed to whatever
motive seems convenient and plausable after the fact. If "predict" and
"anticipate" are not merely distinctions without a difference, the
difference must be that anticipating is a bit less likely to lead to
embarrassment.
> The basic difference between a good and a bad trader would be that,
> given the same information about the market, the good trader understands
> the psychology of the market, and understands how other actors will
> react every time he recieves new pieces of information. A DSP chip can't
> replicate that.
Actually, I don't see why not.
...
Jerry
--
Engineering is the art of making what you want from things you can get.
�����������������������������������������������������������������������
Reply by Rune Allnor●July 31, 20042004-07-31
axlq@spamcop.net (axlq) wrote in message news:<cee658$9ol$1@blue.rahul.net>...
> In article <f56893ae.0407300324.1fcb45fc@posting.google.com>,
> Rune Allnor <allnor@tele.ntnu.no> wrote:
> >axlq@spamcop.net (axlq) wrote in message news:<cecpt1$f99$1@blue.rahul.net>...
> >> So what's the problem? Shed some light please.
> >
> >The problem, as I see it, is that financial data are completely random,
> >while "classical" DSP problems (i.e. applications in communications or
> >physics) have to obey some underlying system.
>
> I guess that would be a problem, if financial data were completely
> random. Completely random financial data would have the same
> statistical character as a random walk -- i.e. the distribution
> of returns are gaussian.
>
> The fact that financial data *doesn't* exhibit this gaussian
> characteristic implies that some determinism is taking place, an
> idea embodied by the Fractal Market Hypothesis (FMH) contradicting
> the traditional Efficient Market Hypothesis (EMH) which posits
> random walks. The fact that the distributions aren't gaussian
> (more like "stable paretian" having a variance of infinity) renders
> worthless all the usual statistical analysis techniques one would
> apply (standard deviation and the like) but that's another subject
> entirely. Anyway, the distribution exhibited by markets implies
> that market trends will last longer than with a random walk, and
> that a market will be prone to large dramatic moves more often.
>
> That said, DSP techniques applied *properly* to financial data --
> say, to remove noise from long-term trends that do exist -- can
> be useful.
I see acontradiction here. A few lines up you say that standard statistical
tools don't apply to financial analysis. Here you say that DSP techniques
can be used for financial analysis. To me, DSP techniques are nothing
more than specially tailored statistical techniques that rely on standard
statistical analysis (mean, standard deviation, variance) and that in
95% of all cases are based on assumptions about the system being Gaussian.
I think you might have answered your own question here.
> The successful commodity traders I have encountered
> don't waste their time trying to make predictions; instead they
> *anticipate* certain behaviors based on past history.
What would be the difference between "predict" and "anticipate"?
The basic difference between a good and a bad trader would be that,
given the same information about the market, the good trader understands
the psychology of the market, and understands how other actors will
react every time he recieves new pieces of information. A DSP chip can't
replicate that.
> DSP simply
> provides some tools with which to build a strategy; DSP isn't *the*
> think these traders use to make decisions.
No it isn't. And it nver will be.
> >One knows, for instance, that a communications link will have
> >certain statistical properties and that these properties will
> >be more or less stable when the link is in use. One knows that
> >certain targets will stay in a combat theatre, and move smoothly
> >under the laws of physics, until they leave or are destroyed. A
> >688 class sub will move differently than an F22 raptor, but both
> >move within their respective physical limits which are constant.
>
> This is where things get interesting, and my field of employment
> (defense) potentially overlaps the financial analysis field.
>
> There are cases where a target is unknown, the dynamics of the
> target cannot be modeled or assumed in advance, and the target is
> capable of deliberate erratic movements with accelerations so great
> that they resemble elastic collisions, AND the sensor tracking the
> target has to contend with positioning errors due to atmospheric
> disturbances and other sources of noise.
>
> This is exactly the problem faced by designers of filters for market
> data, only the financial data is a 1-dimensional time series whereas
> the tracking problem has 2 or 3 dimensions. In fiancial terms, the
> problem could be stated as "how do I track the true long-term price
> movements of this instrument amid all the short-term variations
> (noise), smoothing the trends yet maintaining the high-frequency
> component of the signal at turning points?" The target tracking
> problem is basically the same: "How do I accurately track the motion
> of a target that has occiasional high-frequency components in its
> motion, while filtering out the high-frequency positioning errors?"
This is where you are wrong. You forget the basic premise, that targets
have to obey the laws of physics. Each target will have a top speed and
a minimum (possibly zero or negative) speed. Each target will have a
highest rate of acceleration and rate deceleration. Each target will
have a maximum rate of climbe or decent. Each target will have a
maximum rate of turn. These factors, while perhaps being unknown, are
what the target tracking device are based on. Given an initial position
and velocity, the next position can be estimated based on projecting
the track forward, and adjusting for various track deviations, as well
as observation noise. If you experience problems, there is always the
issue of developing better sensors and collecting better data.
No such thing with the stock market. The data are exact. There are no
fundamental underlying mechanisms for a prediction/anticipaton model.
> Mark Jurik's JMA filter is probably the best thing I've seen in
> the "financial DSP" arena, but it's a proprietary algorithm. In
> my spare time I'm coming up with ways to reverse-engineer it. I
> do know that JMA isn't a spectral filter; it's more distribution
> based, maintaining a measurement of filter error relative to a
> measurement of signal noise and somehow adapting the smoothing
> power accordingly. The result is something that can follow step
> functions with almost no overshoot and yet maintain attenuation of
> the high frequencies that are really present within the signal.
>
> See the noisy sawtooth signal at the bottom of
> http://www.jurikres.com/down/why_jma.pdf and also the noisy
> sawtooth and noisy step function response at the bottom of
> http://www.jurikres.com/down/ma_evolv.pdf -- Pretty cool, if you ask
> me.
>
> -A
Checking out the links will have to wait, as I'm on a dial-up link.
Rune
Reply by axlq●July 31, 20042004-07-31
In article <410ac405$0$2815$61fed72c@news.rcn.com>,
Jerry Avins <jya@ieee.org> wrote:
>dnb is probably aware that Bob has a lively interest in the stock market
>-- he is the moderator of an informal investment group on the web -- and
>I assumed his comment to be of the "in case you're wondering" kind.
Ah. Okay. Chalk it up to me not being familiar with who's who around
here....
-A
Reply by Jerry Avins●July 30, 20042004-07-30
axlq wrote:
> In article <3029ed06.0407301211.77eaa1b6@posting.google.com>,
> Steve <steven_hyh@yahoo.com> wrote:
>
>>Ah, this is an interesting question. Mathmaticicans and computer
>>scientists are using computer modeling to predict the market trend,
>>Economists model the economy trend. The models may be implemented by
>>DSP concepts, eg., lattice filter to do ARMA modeling. That is true
>>the market volitile enough to fail any prediction models. But within
>>certain confidence interval of the prediction, the predictor can make
>>some conclusion with certain success probability. But this is
>>definitely pure DSP or filter issues. Some nonlinear prediction models
>>were used.
>
>
> I think the question I was trying to ask, when I wrote "what's
> the problem?" was more along the lines of, "why do people feel
> the need to add a disclaimer to their articles, stating that
> their application isn't financial?" I get the sense that there's
> some stigma attached to DSP applied to financial analysis, to the
> point where any hint of it is enough to turn people off. Is that
> impression incorrect? I've lurked here only a few weeks but in
> that few weeks I've detected a negative attitude toward financial
> applications.
>
> I can understand that attitude, to a point. DSP methods are often
> poorly understood or mis-applied by people doing technical analysis
> of market data.
>
> MY point (in my other articles in this thread) is that DSP has
> a place in financial analysis, and there's some fertile ground
> there, as long as it's properly used. In my case, I have seen some
> impressive DSP tools invented for financial purposes (and possibly
> mis-applied, but that's shouldn't reflect badly on the invention
> itself), and these tools potentially apply to real-world non-finance
> problems more appropriate for DSP. So what if it originated in
> the financial world? And so what if someone learning about this
> fascinating topic of DSP approaches it from a perspective grounded
> in financial analysis?
>
> -A
dnb is probably aware that Bob has a lively interest in the stock market
-- he is the moderator of an informal investment group on the web -- and
I assumed his comment to be of the "in case you're wondering" kind.
Jerry
--
Engineering is the art of making what you want from things you can get.
�����������������������������������������������������������������������
Reply by axlq●July 30, 20042004-07-30
In article <3029ed06.0407301211.77eaa1b6@posting.google.com>,
Steve <steven_hyh@yahoo.com> wrote:
>Ah, this is an interesting question. Mathmaticicans and computer
>scientists are using computer modeling to predict the market trend,
>Economists model the economy trend. The models may be implemented by
>DSP concepts, eg., lattice filter to do ARMA modeling. That is true
>the market volitile enough to fail any prediction models. But within
>certain confidence interval of the prediction, the predictor can make
>some conclusion with certain success probability. But this is
>definitely pure DSP or filter issues. Some nonlinear prediction models
>were used.
I think the question I was trying to ask, when I wrote "what's
the problem?" was more along the lines of, "why do people feel
the need to add a disclaimer to their articles, stating that
their application isn't financial?" I get the sense that there's
some stigma attached to DSP applied to financial analysis, to the
point where any hint of it is enough to turn people off. Is that
impression incorrect? I've lurked here only a few weeks but in
that few weeks I've detected a negative attitude toward financial
applications.
I can understand that attitude, to a point. DSP methods are often
poorly understood or mis-applied by people doing technical analysis
of market data.
MY point (in my other articles in this thread) is that DSP has
a place in financial analysis, and there's some fertile ground
there, as long as it's properly used. In my case, I have seen some
impressive DSP tools invented for financial purposes (and possibly
mis-applied, but that's shouldn't reflect badly on the invention
itself), and these tools potentially apply to real-world non-finance
problems more appropriate for DSP. So what if it originated in
the financial world? And so what if someone learning about this
fascinating topic of DSP approaches it from a perspective grounded
in financial analysis?
-A
Reply by axlq●July 30, 20042004-07-30
In article <410aa608$0$2818$61fed72c@news.rcn.com>,
Jerry Avins <jya@ieee.org> wrote:
>What assurance does one have that aliasing artifacts are smaller than
>some epsilon? When a single number is recorded once a day at an
>arbitrary time -- closing, say -- what is the upper limit of frequency
>response implied by that?
If you're thinking in terms of spectral analysis, well of course you
can't measure wavelengths smaller than 2 days. However, if you have
used a time series history to characterize the noise in the data
and the error in the filter, one can construct an adaptive filter
that responds quickly to extraordinary events and continues tracking
the "target" with maybe a 1- or 2-day lag. Needless to say, it's
pointless to try to develop a short-term trading strategy with such
a tool, but a long-term strategy based on such a filter might be
worthwhile.
>Certainly, there is information that can be extracted from the numbers.
>I haven't seen a cogent argument to the effect that the methods of DSP
>are good ways to do that.
One way to characterize the high frequency noise in a time series
of prices, if one is interested only in, say trends of 1 month or
longer (about 21 business days), might be to apply a high-pass
filter with a cutoff frequency of 1/21, where the sampling frequency
is 1 day. That's one way I can think of where the methods of DSP
are useful to extract this information.
>> See the noisy sawtooth signal at the bottom of
>> http://www.jurikres.com/down/why_jma.pdf and also the noisy
>> sawtooth and noisy step function response at the bottom of
>> http://www.jurikres.com/down/ma_evolv.pdf -- Pretty cool, if you ask
>> me.
>
>The coolest part is that the low-pass filtering introduces very little
>delay. The only way I could show that is to fake it.
I've seen it working in real time (one of my trader acquaintances
has it as a DLL routine hooked into his software that gathers market
ticks in real time). It's definitely not faked. It really is a
pretty cool filter.
-A