> Mat wrote:
> > Scott Seidman wrote:
> > > "Peter Nachtwey" <peter@deltacompsys.com> wrote in
> > > news:1160198377.424911.108190@m7g2000cwm.googlegroups.com:
> > >
> > > > Jerry Avins wrote:
> > > >> And when you get it working on stock market trends, please let us know.
> > > >
> > > > I am not joking. The market is emotional. Machinery isn't. Search
> > > > for system identification or buy a control book that covers systems
> > > > identification. A good auto tuning program has all of this worked out.
> > > >
> > > > Peter Nachtwey
> >
> > I mention this just to point out that their are those in any domain
> > who find a way to control when to participate during favorable
> > peirod -- yet have negative incentive for public disclosure of any
> > proof. So it is a rather closed viewpoint to believe there are not
> > people with successful models in this area -- just because there
> > is so much noise or because one has spent 10 years or so with
> > no useful solution.
>
>
> > That comes across like -- if I could not do it
> > -- no one can.
> >
> > Mat
Sorry Peter.
I thought that I had signed out gracefully after thanking everyone
and finally engaging in a little unrelated discussion -- that I had
earlier suggested was a distraction.
Recall my post.
"And thank everyone who has experienced a high level of noise in
my description on the problem, reducing it to dsp terms and
your suggestions."
I have been helped my those who contributed to this discussion
and acknowledged as much. I'm already a control specialist
within my domain -- so getting a book is going backwards
because it is sure not to cover what I already know. I don't
mean to sound arrogant -- but I've paid the price to make that
statement. And I will find consultants who can make similar
statements about Kalman filters and Interpolation based on their
skills and efforts and move forward outside of a public forum.
I'm completely content with how this dsp discussion turned out.
Again, thank everyone again for your suggestions and comments.
Mat
> Read Mat. I said get a control book. This has all been done before.
> At least Kalman filters and observers. That was what we were talking
> about then. Now I have no idea what you are talking about and don't
> care. I didn't get a reply to my question so I stopped. You are not
> making is possible for the others to help you even though they are
> trying.
>
> Peter Nachtwey
Reply by Peter Nachtwey●October 9, 20062006-10-09
Mat wrote:
> Scott Seidman wrote:
> > "Peter Nachtwey" <peter@deltacompsys.com> wrote in
> > news:1160198377.424911.108190@m7g2000cwm.googlegroups.com:
> >
> > > Jerry Avins wrote:
> > >> And when you get it working on stock market trends, please let us know.
> > >
> > > I am not joking. The market is emotional. Machinery isn't. Search
> > > for system identification or buy a control book that covers systems
> > > identification. A good auto tuning program has all of this worked out.
> > >
> > > Peter Nachtwey
>
> I mention this just to point out that their are those in any domain
> who find a way to control when to participate during favorable
> peirod -- yet have negative incentive for public disclosure of any
> proof. So it is a rather closed viewpoint to believe there are not
> people with successful models in this area -- just because there
> is so much noise or because one has spent 10 years or so with
> no useful solution.
> That comes across like -- if I could not do it
> -- no one can.
>
> Mat
Read Mat. I said get a control book. This has all been done before.
At least Kalman filters and observers. That was what we were talking
about then. Now I have no idea what you are talking about and don't
care. I didn't get a reply to my question so I stopped. You are not
making is possible for the others to help you even though they are
trying.
Peter Nachtwey
Reply by Scott Seidman●October 9, 20062006-10-09
"Mat" <math_stats@comcast.net> wrote in news:1160400972.214935.84450
@k70g2000cwa.googlegroups.com:
> Maybe trading theory needs to be put on a list with politics and
> religions of subjects to avoid in diverse groups.
On the contrary, high-level economics relies heavily on model-based
control. Think about what the Fed does. They monitor certain indicators,
run them through a model, and adjust interest rates accordingly. Stock
market performance is probably one indicator, and its influenced by the
interest rates.
Of course, not all of the information the Fed would like to have is
observable, the model isn't that good, the system is far from LTI, and they
only have control over one input parameter. That doesn't mean it isn't
control, it just means its hard.
--
Scott
Reverse name to reply
Reply by Mat●October 9, 20062006-10-09
Scott Seidman wrote:
> "Peter Nachtwey" <peter@deltacompsys.com> wrote in
> news:1160198377.424911.108190@m7g2000cwm.googlegroups.com:
>
> > Jerry Avins wrote:
> >> And when you get it working on stock market trends, please let us know.
> >
> > I am not joking. The market is emotional. Machinery isn't. Search
> > for system identification or buy a control book that covers systems
> > identification. A good auto tuning program has all of this worked out.
> >
> > Peter Nachtwey
Maybe trading theory needs to be put on a list with politics and
religions of subjects to avoid in diverse groups. In this century
maybe one of the greatest contributors to a useful theory is
Prof. Thorp. While he cracked the century old problem of the
game of BlackJack and wrote a book about it -- you will find
no such detailed manual of his on trading. Instead he went
straight to the bank and has not written a book on this issue
to change were he is at on the curve.
I mention this just to point out that their are those in any domain
who find a way to control when to participate during favorable
peirod -- yet have negative incentive for public disclosure of any
proof. So it is a rather closed viewpoint to believe there are not
people with successful models in this area -- just because there
is so much noise or because one has spent 10 years or so with
no useful solution. That comes across like -- if I could not do it
-- no one can.
Mat
> The whole point of sys ID is that you don't need a GOOD model of the
> system, just a model good enough to control the system, and you can let the
> parameters drift as the data comes in to capture higher order poles and the
> like.
>
> --
> Scott
> Reverse name to reply
Reply by Mat●October 9, 20062006-10-09
Jerry Avins wrote:
> Mat wrote:
>
> ...
>
> > If you do a google search on "predict the smoothed filter" or similiar
> > wording you will find that when the raw value are know -- that it's
> > possible to predict their smoothed values. Take a look at the
> > background of people in this area and you will find they are using
> > mathematical models and not crystal balls. It looks like most of
> > this work is going on outside of the DSP field -- so I can
> > understand if the idea seems foriegn.
Only after your comments below could I understand your meaning.
I just point out for the record -- that in the context of your previous
assumption -- "crystal ball" means he wants to know the future.
This ain't so. While its clear "Interpolation" and "Extrapolation" have
different meanings and my terminology of predicting the smoothed
mean or filter are far from ideal -- the series of replies that I have
made during this post consistently indicate my goal was to derive
a lag free Interpolation process. Since I only trust control methods
in my domain -- even such a process would fit into a control
scheme.
Now that I know your usage of this term -- I'm able to view it
differently. Thank you for clearly things up.
And thank everyone who has experienced a high level of noise in
my description on the problem, reducing it to dsp terms and
your suggestions.
Mat
> I hope you understand that by "crystal ball" I meant "method of
> prediction" without disparagement. One fine beginner's book on DSP
> defines a pole as (1) a root of the denominator of a response equation
> or (2) a box filled with magic powder that can do wonderful things, and
> adds that most practicing engineers most of the time use (2).
>
> Jerry
> --
> "The rights of the best of men are secured only as the
> rights of the vilest and most abhorrent are protected."
> - Chief Justice Charles Evans Hughes, 1927
> =AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=AF=
"Peter Nachtwey" <peter@deltacompsys.com> wrote in
news:1160198377.424911.108190@m7g2000cwm.googlegroups.com:
> Jerry Avins wrote:
>> And when you get it working on stock market trends, please let us know.
>
> I am not joking. The market is emotional. Machinery isn't. Search
> for system identification or buy a control book that covers systems
> identification. A good auto tuning program has all of this worked out.
>
> Peter Nachtwey
>
The whole point of sys ID is that you don't need a GOOD model of the
system, just a model good enough to control the system, and you can let the
parameters drift as the data comes in to capture higher order poles and the
like.
--
Scott
Reverse name to reply
Reply by Jerry Avins●October 8, 20062006-10-08
Mat wrote:
...
> If you do a google search on "predict the smoothed filter" or similiar
> wording you will find that when the raw value are know -- that it's
> possible to predict their smoothed values. Take a look at the
> background of people in this area and you will find they are using
> mathematical models and not crystal balls. It looks like most of
> this work is going on outside of the DSP field -- so I can
> understand if the idea seems foriegn.
I hope you understand that by "crystal ball" I meant "method of
prediction" without disparagement. One fine beginner's book on DSP
defines a pole as (1) a root of the denominator of a response equation
or (2) a box filled with magic powder that can do wonderful things, and
adds that most practicing engineers most of the time use (2).
Jerry
--
"The rights of the best of men are secured only as the
rights of the vilest and most abhorrent are protected."
- Chief Justice Charles Evans Hughes, 1927
���������������������������������������������������������������������
Reply by Mat●October 8, 20062006-10-08
jim wrote:
> Mat wrote:
> >
> > jim wrote:
> > > Jerry Avins wrote:
> > >
> > > >
> > > > The output of the filter is delayed by some amount that depends on the
> > > > filter. Some filters are more "prompt" than linear-phase filters.
> > > > (Minimum-phase filters are the most prompt of all.) Mat wants to predict
> > > > when the delay begins what the output is likely to be after the delay
> > > > period has elapsed.
> >
> > The following url has an explanation for kriging.
> > http://en.wikipedia.org/wiki/Kriging
> >
> > And more importantly it show a graph of one-dimensional data
> > interpolation by Kriging, Following is the explanation they provide.
> >
> > "A set of values are then observed, each value associated with a
> > spatial location. Now, a new value can be predicted at any new spatial
> > location, by combining the Gaussian prior with a Gaussian likelihood
> > function for each of the observed values. The resulting posterior
> > distribution is also a Gaussian, with a mean and covariance that can be
> > simply computed from the observed values, "
>
>
> So the problem is that you have a sparsely populated data set and you
> want to be able to fill in the missing data by creating a smooth
> function that passes thru the existing data and can be resampled at any
> point you like. That is nothing at all like the problem Jerry described.
> I was responding to his description of the problem not yours.
>
>
> >
> > Now I have a regression consultant who will soon try to explore whether
> > his techniques will be useful on the solution that been posed and
> > rephrased several times. If Kalman is more practical than the two
> > interpolation methods mentioned several times I'll find a consultant
> > in this area -- but if interpolation is more durable that will be my
> > next direction. So at this point I'm looking very broadly to find
> > the modeling solution others are using for the task described. My
> > regression consultant is not involved with Gaussian Process
> > Regression which I understand is a more generalized form related
> > to Kriging.
You are right. Try predicting smoothed mean, predicting smoothed
filter, predicted smoothed -- and you will see results as well as with
using terms associated with this concept ( leave quotes off and surely
this cannot be hard to find).
> > If you do a google search on "predict the smoothed filter" or similiar
> > wording you will find that when the raw value are know -- that it's
> > possible to predict their smoothed values.
>
> Your quoted text produces no results from Google. It seems that by
> "filter" you actually mean function. That is, you are attempting to
> create a smooth function that passes thru your data points.
Yes, I think your comments are "on the money". My data is at
regular intervals, has restrained bounds and is a single variable
controlling the whole operation. If i'm able to develop a useful
prediction of smoothed from raw data using Spline interpolation
I may be able to rely less on my expectations which are useful
but not a accurate as my variable.
The two interpolation methods that I mentioned are a variant of
Spline interpolation.
Mat
> Interpolation sounds like what you want to do. Spline interpolation is
> probably the most widely used method for doing this. Particularly when
> the data is not at regular intervals. Without knowing anymore about what
> your exact requirements might, I would think that spline interpolation
> would be the first thing you would attempt to use.
>
> -jim
>
>
> >Take a look at the
> > background of people in this area and you will find they are using
> > mathematical models and not crystal balls. It looks like most of
> > this work is going on outside of the DSP field -- so I can
> > understand if the idea seems foriegn.
> >
> > Mat
> >
> > > What do you mean "predict"? Is someone going to design this filter? Does
> > > he want to know where he can find a crystal ball that will tell him what
> > > the filter this person is going to design will look like?
> > >
> > > -jim
> > >
> > > ----== Posted via Newsfeeds.Com - Unlimited-Unrestricted-Secure Usenet News==----
> > > http://www.newsfeeds.com The #1 Newsgroup Service in the World! 120,000+ Newsgroups
> > > ----= East and West-Coast Server Farms - Total Privacy via Encryption =----
>
> ----== Posted via Newsfeeds.Com - Unlimited-Unrestricted-Secure Usenet News==----
> http://www.newsfeeds.com The #1 Newsgroup Service in the World! 120,000+ Newsgroups
> ----= East and West-Coast Server Farms - Total Privacy via Encryption =----
Reply by Mat●October 8, 20062006-10-08
Jerry Wolf wrote:
> On Oct 6, 4:03 pm, "Mat" <math_st...@comcast.net> wrote:
> > It may have been better to say that I want to use my raw data to
> > predict the smoothed trend without any lag.
My signal duration at various levels is robust. Up to a point I
can act on the expectancies that generally occur while the signal
is useful. The expectancies are derived using simple moving
average so I feel more comfortable staying with the same filter that
has generated information that is occurring over and over with low
variance. I know the data and if expectancies are outside of
my tolerance control level -- the expectancies subsume the
signal.
I would have no use in exponential, etc. for the above reasons.
> Maybe I'm missing the point, but why don't you consider computing a
> smoothed trend from only the past samples, as in an exponential window
> going back into the past? That has zero lag.
>
> > So I would like to
> > know if I could use something like a Kalman filter to develop
> > a model to do this on say 1000 to 5000 data points which
> > would have enough predictive powers to predict the smoothed
> > values using my raw data. I know prediction issues are often
> > dealt with from the statistical viewpoint using regression and
> > similar models.
> >
> > Can Kalman or some other dsp algorithm be trained on
> > raw data to predict filtered values with zero lag?
Reply by jim●October 8, 20062006-10-08
Mat wrote:
>
> jim wrote:
> > Jerry Avins wrote:
> >
> > >
> > > The output of the filter is delayed by some amount that depends on the
> > > filter. Some filters are more "prompt" than linear-phase filters.
> > > (Minimum-phase filters are the most prompt of all.) Mat wants to predict
> > > when the delay begins what the output is likely to be after the delay
> > > period has elapsed.
>
> The following url has an explanation for kriging.
> http://en.wikipedia.org/wiki/Kriging
>
> And more importantly it show a graph of one-dimensional data
> interpolation by Kriging, Following is the explanation they provide.
>
> "A set of values are then observed, each value associated with a
> spatial location. Now, a new value can be predicted at any new spatial
> location, by combining the Gaussian prior with a Gaussian likelihood
> function for each of the observed values. The resulting posterior
> distribution is also a Gaussian, with a mean and covariance that can be
> simply computed from the observed values, "
So the problem is that you have a sparsely populated data set and you
want to be able to fill in the missing data by creating a smooth
function that passes thru the existing data and can be resampled at any
point you like. That is nothing at all like the problem Jerry described.
I was responding to his description of the problem not yours.
>
> Now I have a regression consultant who will soon try to explore whether
> his techniques will be useful on the solution that been posed and
> rephrased several times. If Kalman is more practical than the two
> interpolation methods mentioned several times I'll find a consultant
> in this area -- but if interpolation is more durable that will be my
> next direction. So at this point I'm looking very broadly to find
> the modeling solution others are using for the task described. My
> regression consultant is not involved with Gaussian Process
> Regression which I understand is a more generalized form related
> to Kriging.
>
> If you do a google search on "predict the smoothed filter" or similiar
> wording you will find that when the raw value are know -- that it's
> possible to predict their smoothed values.
Your quoted text produces no results from Google. It seems that by
"filter" you actually mean function. That is, you are attempting to
create a smooth function that passes thru your data points.
Interpolation sounds like what you want to do. Spline interpolation is
probably the most widely used method for doing this. Particularly when
the data is not at regular intervals. Without knowing anymore about what
your exact requirements might, I would think that spline interpolation
would be the first thing you would attempt to use.
-jim
>Take a look at the
> background of people in this area and you will find they are using
> mathematical models and not crystal balls. It looks like most of
> this work is going on outside of the DSP field -- so I can
> understand if the idea seems foriegn.
>
> Mat
>
> > What do you mean "predict"? Is someone going to design this filter? Does
> > he want to know where he can find a crystal ball that will tell him what
> > the filter this person is going to design will look like?
> >
> > -jim
> >
> > ----== Posted via Newsfeeds.Com - Unlimited-Unrestricted-Secure Usenet News==----
> > http://www.newsfeeds.com The #1 Newsgroup Service in the World! 120,000+ Newsgroups
> > ----= East and West-Coast Server Farms - Total Privacy via Encryption =----
----== Posted via Newsfeeds.Com - Unlimited-Unrestricted-Secure Usenet News==----
http://www.newsfeeds.com The #1 Newsgroup Service in the World! 120,000+ Newsgroups
----= East and West-Coast Server Farms - Total Privacy via Encryption =----