In article <410a651f$0$2815$61fed72c@news.rcn.com>, Jerry Avins <jya@ieee.org> wrote:>I wrote: >> So what's the problem? Shed some light please. > >I happen to think that the fate of companies and the worth of their >stock depends more on how they are managed and other real-world events >far more than past history of ups and downs in price.I agree, but technical analysis applies to more than stocks. I personally know some wealthy commodity futures traders who make a living trading these instruments, using technical analysis (and no, they don't make a living selling books or seminars, they trade privately for their families). I have observed that commodity prices, while influenced by real world events, do seem to embody some fundamental relationships in past price movements. Granted, these relationship are based more in probability distributions, fractals, fibonacci series, geometry, and the like, and less so on spectral analysis. However, these successful traders DO use filters to elminate noise or detrend data when needed.>"Technical analysis" tries to predict future prices by finding >trends in past prices, with little or no attention to "outside" >variables. I think it's a crock.The people who do this successfully don't try to predict future prices at all. They have told me that doing this is a crock -- they would agree with you. They anticipate future movements based on past history, with fallback positions for being incorrect. Anticipation does not equal prediction. -A
Designing Filters with non-uniform sampling
Started by ●July 27, 2004
Reply by ●July 30, 20042004-07-30
Reply by ●July 30, 20042004-07-30
In article <f56893ae.0407300324.1fcb45fc@posting.google.com>, Rune Allnor <allnor@tele.ntnu.no> wrote:>axlq@spamcop.net (axlq) wrote in message news:<cecpt1$f99$1@blue.rahul.net>... >> So what's the problem? Shed some light please. > >The problem, as I see it, is that financial data are completely random, >while "classical" DSP problems (i.e. applications in communications or >physics) have to obey some underlying system.I guess that would be a problem, if financial data were completely random. Completely random financial data would have the same statistical character as a random walk -- i.e. the distribution of returns are gaussian. The fact that financial data *doesn't* exhibit this gaussian characteristic implies that some determinism is taking place, an idea embodied by the Fractal Market Hypothesis (FMH) contradicting the traditional Efficient Market Hypothesis (EMH) which posits random walks. The fact that the distributions aren't gaussian (more like "stable paretian" having a variance of infinity) renders worthless all the usual statistical analysis techniques one would apply (standard deviation and the like) but that's another subject entirely. Anyway, the distribution exhibited by markets implies that market trends will last longer than with a random walk, and that a market will be prone to large dramatic moves more often. That said, DSP techniques applied *properly* to financial data -- say, to remove noise from long-term trends that do exist -- can be useful. The successful commodity traders I have encountered don't waste their time trying to make predictions; instead they *anticipate* certain behaviors based on past history. DSP simply provides some tools with which to build a strategy; DSP isn't *the* think these traders use to make decisions.>One knows, for instance, that a communications link will have >certain statistical properties and that these properties will >be more or less stable when the link is in use. One knows that >certain targets will stay in a combat theatre, and move smoothly >under the laws of physics, until they leave or are destroyed. A >688 class sub will move differently than an F22 raptor, but both >move within their respective physical limits which are constant.This is where things get interesting, and my field of employment (defense) potentially overlaps the financial analysis field. There are cases where a target is unknown, the dynamics of the target cannot be modeled or assumed in advance, and the target is capable of deliberate erratic movements with accelerations so great that they resemble elastic collisions, AND the sensor tracking the target has to contend with positioning errors due to atmospheric disturbances and other sources of noise. This is exactly the problem faced by designers of filters for market data, only the financial data is a 1-dimensional time series whereas the tracking problem has 2 or 3 dimensions. In fiancial terms, the problem could be stated as "how do I track the true long-term price movements of this instrument amid all the short-term variations (noise), smoothing the trends yet maintaining the high-frequency component of the signal at turning points?" The target tracking problem is basically the same: "How do I accurately track the motion of a target that has occiasional high-frequency components in its motion, while filtering out the high-frequency positioning errors?" Mark Jurik's JMA filter is probably the best thing I've seen in the "financial DSP" arena, but it's a proprietary algorithm. In my spare time I'm coming up with ways to reverse-engineer it. I do know that JMA isn't a spectral filter; it's more distribution based, maintaining a measurement of filter error relative to a measurement of signal noise and somehow adapting the smoothing power accordingly. The result is something that can follow step functions with almost no overshoot and yet maintain attenuation of the high frequencies that are really present within the signal. See the noisy sawtooth signal at the bottom of http://www.jurikres.com/down/why_jma.pdf and also the noisy sawtooth and noisy step function response at the bottom of http://www.jurikres.com/down/ma_evolv.pdf -- Pretty cool, if you ask me. -A
Reply by ●July 30, 20042004-07-30
axlq wrote:> In article <410a651f$0$2815$61fed72c@news.rcn.com>, > Jerry Avins <jya@ieee.org> wrote: > >>I wrote: >> >>>So what's the problem? Shed some light please. >> >>I happen to think that the fate of companies and the worth of their >>stock depends more on how they are managed and other real-world events >>far more than past history of ups and downs in price. > > > I agree, but technical analysis applies to more than stocks. I > personally know some wealthy commodity futures traders who make a > living trading these instruments, using technical analysis (and > no, they don't make a living selling books or seminars, they trade > privately for their families). I have observed that commodity > prices, while influenced by real world events, do seem to embody > some fundamental relationships in past price movements. Granted, > these relationship are based more in probability distributions, > fractals, fibonacci series, geometry, and the like, and less so on > spectral analysis. However, these successful traders DO use filters > to elminate noise or detrend data when needed. > > >>"Technical analysis" tries to predict future prices by finding >>trends in past prices, with little or no attention to "outside" >>variables. I think it's a crock. > > > The people who do this successfully don't try to predict future > prices at all. They have told me that doing this is a crock -- > they would agree with you. They anticipate future movements based > on past history, with fallback positions for being incorrect. > Anticipation does not equal prediction. > > -AI hear you. I can make a good argument for the empirical value of astrology along similar lines. We live in a climate that makes living conditions very dependent on the season. Almost everyone agrees that early formative experiences influence the way we conduct out entire lives. Someone beginning to toddle in a clement season, when wandering indoors or out is unrestrained and easy develops in an environment that's entirely different from someone born months earlier or later. Having to be heavily swaddled when outdoors certainly changes the experience, especially before a steady supply of clean diapers could be taken for granted. Astrologies rarely develop or flourish in tropical climates. Some price trends are real and predictable to some extent. (Acting on an anticipation may differ semantically from acting on a prediction, but it differs little if at all practically.) The problem is knowing which ones. It's a little like those faith-healing stories. We shouldn't be surprised to hear that someone trusted in his faith and was [rescued, healed, saved; pick one]. Many people trust their faiths to rescue them. We hear only from the survivors, a necessarily biased sample. Jerry -- Engineering is the art of making what you want from things you can get. �����������������������������������������������������������������������
Reply by ●July 30, 20042004-07-30
axlq wrote:> In article <f56893ae.0407300324.1fcb45fc@posting.google.com>, > Rune Allnor <allnor@tele.ntnu.no> wrote: > >>axlq@spamcop.net (axlq) wrote in message news:<cecpt1$f99$1@blue.rahul.net>... >> >>>So what's the problem? Shed some light please. >> >>The problem, as I see it, is that financial data are completely random, >>while "classical" DSP problems (i.e. applications in communications or >>physics) have to obey some underlying system. > > > I guess that would be a problem, if financial data were completely > random. Completely random financial data would have the same > statistical character as a random walk -- i.e. the distribution > of returns are gaussian. > > The fact that financial data *doesn't* exhibit this gaussian > characteristic implies that some determinism is taking place, an > idea embodied by the Fractal Market Hypothesis (FMH) contradicting > the traditional Efficient Market Hypothesis (EMH) which posits > random walks. The fact that the distributions aren't gaussian > (more like "stable paretian" having a variance of infinity) renders > worthless all the usual statistical analysis techniques one would > apply (standard deviation and the like) but that's another subject > entirely. Anyway, the distribution exhibited by markets implies > that market trends will last longer than with a random walk, and > that a market will be prone to large dramatic moves more often. > > That said, DSP techniques applied *properly* to financial data -- > say, to remove noise from long-term trends that do exist -- can > be useful. The successful commodity traders I have encountered > don't waste their time trying to make predictions; instead they > *anticipate* certain behaviors based on past history. DSP simply > provides some tools with which to build a strategy; DSP isn't *the* > think these traders use to make decisions.What assurance does one have that aliasing artifacts are smaller than some epsilon? When a single number is recorded once a day at an arbitrary time -- closing, say -- what is the upper limit of frequency response implied by that? Certainly, there is information that can be extracted from the numbers. I haven't seen a cogent argument to the effect that the methods of DSP are good ways to do that.>>One knows, for instance, that a communications link will have >>certain statistical properties and that these properties will >>be more or less stable when the link is in use. One knows that >>certain targets will stay in a combat theatre, and move smoothly >>under the laws of physics, until they leave or are destroyed. A >>688 class sub will move differently than an F22 raptor, but both >>move within their respective physical limits which are constant. > > > This is where things get interesting, and my field of employment > (defense) potentially overlaps the financial analysis field. > > There are cases where a target is unknown, the dynamics of the > target cannot be modeled or assumed in advance, and the target is > capable of deliberate erratic movements with accelerations so great > that they resemble elastic collisions, AND the sensor tracking the > target has to contend with positioning errors due to atmospheric > disturbances and other sources of noise. > > This is exactly the problem faced by designers of filters for market > data, only the financial data is a 1-dimensional time series whereas > the tracking problem has 2 or 3 dimensions. In fiancial terms, the > problem could be stated as "how do I track the true long-term price > movements of this instrument amid all the short-term variations > (noise), smoothing the trends yet maintaining the high-frequency > component of the signal at turning points?" The target tracking > problem is basically the same: "How do I accurately track the motion > of a target that has occiasional high-frequency components in its > motion, while filtering out the high-frequency positioning errors?" > > Mark Jurik's JMA filter is probably the best thing I've seen in > the "financial DSP" arena, but it's a proprietary algorithm. In > my spare time I'm coming up with ways to reverse-engineer it. I > do know that JMA isn't a spectral filter; it's more distribution > based, maintaining a measurement of filter error relative to a > measurement of signal noise and somehow adapting the smoothing > power accordingly. The result is something that can follow step > functions with almost no overshoot and yet maintain attenuation of > the high frequencies that are really present within the signal. > > See the noisy sawtooth signal at the bottom of > http://www.jurikres.com/down/why_jma.pdf and also the noisy > sawtooth and noisy step function response at the bottom of > http://www.jurikres.com/down/ma_evolv.pdf -- Pretty cool, if you ask > me.The coolest part is that the low-pass filtering introduces very little delay. The only way I could show that is to fake it. Jerry -- Engineering is the art of making what you want from things you can get. �����������������������������������������������������������������������
Reply by ●July 30, 20042004-07-30
Ah, this is an interesting question. Mathmaticicans and computer scientists are using computer modeling to predict the market trend, Economists model the economy trend. The models may be implemented by DSP concepts, eg., lattice filter to do ARMA modeling. That is true the market volitile enough to fail any prediction models. But within certain confidence interval of the prediction, the predictor can make some conclusion with certain success probability. But this is definitely pure DSP or filter issues. Some nonlinear prediction models were used. axlq@spamcop.net (axlq) wrote in message news:<cecpt1$f99$1@blue.rahul.net>...> In article <fa0deac8.0407290559.66cc0911@posting.google.com>, > dnb <dbhat2@yahoo.com> wrote: > >Thanks much for the reply. Bob - this is not a project for Wall > >Street, and I do not work in the financial world! > > I've been lurking here for a while, and even posted a few times. I > am curious about the statement above, because I have seen similar > sentiments expressed in other posts. What is it about DSP applied > to financial analysis that seems to turn folks off here? Me, my > background is physics and I work in the defense industry, but I do > find it interesting to invent filters and apply them to market data > problems; in fact much of what I learned about digital filters for > tracking targets in a military context, came from looking at such > filters applied to market movements. > > So what's the problem? Shed some light please. > > -A
Reply by ●July 30, 20042004-07-30
In article <410aa608$0$2818$61fed72c@news.rcn.com>, Jerry Avins <jya@ieee.org> wrote:>What assurance does one have that aliasing artifacts are smaller than >some epsilon? When a single number is recorded once a day at an >arbitrary time -- closing, say -- what is the upper limit of frequency >response implied by that?If you're thinking in terms of spectral analysis, well of course you can't measure wavelengths smaller than 2 days. However, if you have used a time series history to characterize the noise in the data and the error in the filter, one can construct an adaptive filter that responds quickly to extraordinary events and continues tracking the "target" with maybe a 1- or 2-day lag. Needless to say, it's pointless to try to develop a short-term trading strategy with such a tool, but a long-term strategy based on such a filter might be worthwhile.>Certainly, there is information that can be extracted from the numbers. >I haven't seen a cogent argument to the effect that the methods of DSP >are good ways to do that.One way to characterize the high frequency noise in a time series of prices, if one is interested only in, say trends of 1 month or longer (about 21 business days), might be to apply a high-pass filter with a cutoff frequency of 1/21, where the sampling frequency is 1 day. That's one way I can think of where the methods of DSP are useful to extract this information.>> See the noisy sawtooth signal at the bottom of >> http://www.jurikres.com/down/why_jma.pdf and also the noisy >> sawtooth and noisy step function response at the bottom of >> http://www.jurikres.com/down/ma_evolv.pdf -- Pretty cool, if you ask >> me. > >The coolest part is that the low-pass filtering introduces very little >delay. The only way I could show that is to fake it.I've seen it working in real time (one of my trader acquaintances has it as a DLL routine hooked into his software that gathers market ticks in real time). It's definitely not faked. It really is a pretty cool filter. -A
Reply by ●July 30, 20042004-07-30
In article <3029ed06.0407301211.77eaa1b6@posting.google.com>, Steve <steven_hyh@yahoo.com> wrote:>Ah, this is an interesting question. Mathmaticicans and computer >scientists are using computer modeling to predict the market trend, >Economists model the economy trend. The models may be implemented by >DSP concepts, eg., lattice filter to do ARMA modeling. That is true >the market volitile enough to fail any prediction models. But within >certain confidence interval of the prediction, the predictor can make >some conclusion with certain success probability. But this is >definitely pure DSP or filter issues. Some nonlinear prediction models >were used.I think the question I was trying to ask, when I wrote "what's the problem?" was more along the lines of, "why do people feel the need to add a disclaimer to their articles, stating that their application isn't financial?" I get the sense that there's some stigma attached to DSP applied to financial analysis, to the point where any hint of it is enough to turn people off. Is that impression incorrect? I've lurked here only a few weeks but in that few weeks I've detected a negative attitude toward financial applications. I can understand that attitude, to a point. DSP methods are often poorly understood or mis-applied by people doing technical analysis of market data. MY point (in my other articles in this thread) is that DSP has a place in financial analysis, and there's some fertile ground there, as long as it's properly used. In my case, I have seen some impressive DSP tools invented for financial purposes (and possibly mis-applied, but that's shouldn't reflect badly on the invention itself), and these tools potentially apply to real-world non-finance problems more appropriate for DSP. So what if it originated in the financial world? And so what if someone learning about this fascinating topic of DSP approaches it from a perspective grounded in financial analysis? -A
Reply by ●July 30, 20042004-07-30
axlq wrote:> In article <3029ed06.0407301211.77eaa1b6@posting.google.com>, > Steve <steven_hyh@yahoo.com> wrote: > >>Ah, this is an interesting question. Mathmaticicans and computer >>scientists are using computer modeling to predict the market trend, >>Economists model the economy trend. The models may be implemented by >>DSP concepts, eg., lattice filter to do ARMA modeling. That is true >>the market volitile enough to fail any prediction models. But within >>certain confidence interval of the prediction, the predictor can make >>some conclusion with certain success probability. But this is >>definitely pure DSP or filter issues. Some nonlinear prediction models >>were used. > > > I think the question I was trying to ask, when I wrote "what's > the problem?" was more along the lines of, "why do people feel > the need to add a disclaimer to their articles, stating that > their application isn't financial?" I get the sense that there's > some stigma attached to DSP applied to financial analysis, to the > point where any hint of it is enough to turn people off. Is that > impression incorrect? I've lurked here only a few weeks but in > that few weeks I've detected a negative attitude toward financial > applications. > > I can understand that attitude, to a point. DSP methods are often > poorly understood or mis-applied by people doing technical analysis > of market data. > > MY point (in my other articles in this thread) is that DSP has > a place in financial analysis, and there's some fertile ground > there, as long as it's properly used. In my case, I have seen some > impressive DSP tools invented for financial purposes (and possibly > mis-applied, but that's shouldn't reflect badly on the invention > itself), and these tools potentially apply to real-world non-finance > problems more appropriate for DSP. So what if it originated in > the financial world? And so what if someone learning about this > fascinating topic of DSP approaches it from a perspective grounded > in financial analysis? > > -Adnb is probably aware that Bob has a lively interest in the stock market -- he is the moderator of an informal investment group on the web -- and I assumed his comment to be of the "in case you're wondering" kind. Jerry -- Engineering is the art of making what you want from things you can get. �����������������������������������������������������������������������
Reply by ●July 31, 20042004-07-31
In article <410ac405$0$2815$61fed72c@news.rcn.com>, Jerry Avins <jya@ieee.org> wrote:>dnb is probably aware that Bob has a lively interest in the stock market >-- he is the moderator of an informal investment group on the web -- and >I assumed his comment to be of the "in case you're wondering" kind.Ah. Okay. Chalk it up to me not being familiar with who's who around here.... -A
Reply by ●July 31, 20042004-07-31
axlq@spamcop.net (axlq) wrote in message news:<cee658$9ol$1@blue.rahul.net>...> In article <f56893ae.0407300324.1fcb45fc@posting.google.com>, > Rune Allnor <allnor@tele.ntnu.no> wrote: > >axlq@spamcop.net (axlq) wrote in message news:<cecpt1$f99$1@blue.rahul.net>... > >> So what's the problem? Shed some light please. > > > >The problem, as I see it, is that financial data are completely random, > >while "classical" DSP problems (i.e. applications in communications or > >physics) have to obey some underlying system. > > I guess that would be a problem, if financial data were completely > random. Completely random financial data would have the same > statistical character as a random walk -- i.e. the distribution > of returns are gaussian. > > The fact that financial data *doesn't* exhibit this gaussian > characteristic implies that some determinism is taking place, an > idea embodied by the Fractal Market Hypothesis (FMH) contradicting > the traditional Efficient Market Hypothesis (EMH) which posits > random walks. The fact that the distributions aren't gaussian > (more like "stable paretian" having a variance of infinity) renders > worthless all the usual statistical analysis techniques one would > apply (standard deviation and the like) but that's another subject > entirely. Anyway, the distribution exhibited by markets implies > that market trends will last longer than with a random walk, and > that a market will be prone to large dramatic moves more often. > > That said, DSP techniques applied *properly* to financial data -- > say, to remove noise from long-term trends that do exist -- can > be useful.I see acontradiction here. A few lines up you say that standard statistical tools don't apply to financial analysis. Here you say that DSP techniques can be used for financial analysis. To me, DSP techniques are nothing more than specially tailored statistical techniques that rely on standard statistical analysis (mean, standard deviation, variance) and that in 95% of all cases are based on assumptions about the system being Gaussian. I think you might have answered your own question here.> The successful commodity traders I have encountered > don't waste their time trying to make predictions; instead they > *anticipate* certain behaviors based on past history.What would be the difference between "predict" and "anticipate"? The basic difference between a good and a bad trader would be that, given the same information about the market, the good trader understands the psychology of the market, and understands how other actors will react every time he recieves new pieces of information. A DSP chip can't replicate that.> DSP simply > provides some tools with which to build a strategy; DSP isn't *the* > think these traders use to make decisions.No it isn't. And it nver will be.> >One knows, for instance, that a communications link will have > >certain statistical properties and that these properties will > >be more or less stable when the link is in use. One knows that > >certain targets will stay in a combat theatre, and move smoothly > >under the laws of physics, until they leave or are destroyed. A > >688 class sub will move differently than an F22 raptor, but both > >move within their respective physical limits which are constant. > > This is where things get interesting, and my field of employment > (defense) potentially overlaps the financial analysis field. > > There are cases where a target is unknown, the dynamics of the > target cannot be modeled or assumed in advance, and the target is > capable of deliberate erratic movements with accelerations so great > that they resemble elastic collisions, AND the sensor tracking the > target has to contend with positioning errors due to atmospheric > disturbances and other sources of noise. > > This is exactly the problem faced by designers of filters for market > data, only the financial data is a 1-dimensional time series whereas > the tracking problem has 2 or 3 dimensions. In fiancial terms, the > problem could be stated as "how do I track the true long-term price > movements of this instrument amid all the short-term variations > (noise), smoothing the trends yet maintaining the high-frequency > component of the signal at turning points?" The target tracking > problem is basically the same: "How do I accurately track the motion > of a target that has occiasional high-frequency components in its > motion, while filtering out the high-frequency positioning errors?"This is where you are wrong. You forget the basic premise, that targets have to obey the laws of physics. Each target will have a top speed and a minimum (possibly zero or negative) speed. Each target will have a highest rate of acceleration and rate deceleration. Each target will have a maximum rate of climbe or decent. Each target will have a maximum rate of turn. These factors, while perhaps being unknown, are what the target tracking device are based on. Given an initial position and velocity, the next position can be estimated based on projecting the track forward, and adjusting for various track deviations, as well as observation noise. If you experience problems, there is always the issue of developing better sensors and collecting better data. No such thing with the stock market. The data are exact. There are no fundamental underlying mechanisms for a prediction/anticipaton model.> Mark Jurik's JMA filter is probably the best thing I've seen in > the "financial DSP" arena, but it's a proprietary algorithm. In > my spare time I'm coming up with ways to reverse-engineer it. I > do know that JMA isn't a spectral filter; it's more distribution > based, maintaining a measurement of filter error relative to a > measurement of signal noise and somehow adapting the smoothing > power accordingly. The result is something that can follow step > functions with almost no overshoot and yet maintain attenuation of > the high frequencies that are really present within the signal. > > See the noisy sawtooth signal at the bottom of > http://www.jurikres.com/down/why_jma.pdf and also the noisy > sawtooth and noisy step function response at the bottom of > http://www.jurikres.com/down/ma_evolv.pdf -- Pretty cool, if you ask > me. > > -AChecking out the links will have to wait, as I'm on a dial-up link. Rune
