How to interpolate this curve?

How to interpolate this curve?

Hi all,

Please take a look at the following plot:

http://img63.imageshack.us/img63/5599/gggyf5.jpg

There are two curves:

The red one is the ideal one and the blue one is the real data from 
experiments.

Due to some numerical instability, the real data strays away from the ideal 
curve at around x=15, and then begins oscillates weirdly after x=19. Since 
the real data is definitely wrong beyond x=19, I only capture the real data 
from x=1 to x=19.

I also know that at x=+infinity, the asymptotic value of the curve should be 
1.5. That's to say, the ideal curve slowly converges to y=1.5 from below and 
in theory it never touches y=1.5, but numerically, it can reach y=1.5 when x 
is a very large number.

The questions is how to recover the ideal curve(the red one) from the 
portion of the real data(the blue curve, up to x=19). More specifically:

1. Is there a symtematic method to detect the turning point starting from 
which the real curve begins to stray away from the ideal curve (here is 
x=15)?

If I throw away the data from x=15 to x=19, and only do the interpolaton 
based on the data from x=1 to x=15. Then I will probably do a very good job 
fitting the curve. However, I want to find a systematic method that works 
automatically and programmatically for all such curves but with varying 
turning points. It's going to be troublesome if every time I have to 
physically inspect the curves use my eyes manually.

2. Is there a way to utilize the knowledge of the asymptotic value y=1.5?
I tried to do polynomial fit and other cubic line fit for x=1 to 19 and then 
x=100000(at which y=1.5), but the result is not very good.

There must be a way to exploit the smoothness of the curve and recover the 
ideal curve(the red one) based on the partial real data(the blue curve, up 
to x=19, before it diverges)...

Please shed some lights on me! Thanks a lot!

On Jul 10, 11:18 pm, "Linus Utopia" <linus_uto...@gmail.com> wrote:
> How to interpolate this curve?
>
> Hi all,
>
> Please take a look at the following plot:
>
> http://img63.imageshack.us/img63/5599/gggyf5.jpg
>
> There are two curves:
>
> The red one is the ideal one and the blue one is the real data from
> experiments.
>
> Due to some numerical instability, the real data strays away from the ideal
> curve at around x=15, and then begins oscillates weirdly after x=19. Since
> the real data is definitely wrong beyond x=19, I only capture the real data
> from x=1 to x=19.
>
> I also know that at x=+infinity, the asymptotic value of the curve should be
> 1.5. That's to say, the ideal curve slowly converges to y=1.5 from below and
> in theory it never touches y=1.5, but numerically, it can reach y=1.5 when x
> is a very large number.
>
> The questions is how to recover the ideal curve(the red one) from the
> portion of the real data(the blue curve, up to x=19). More specifically:
>
> 1. Is there a symtematic method to detect the turning point starting from
> which the real curve begins to stray away from the ideal curve (here is
> x=15)?
>
> If I throw away the data from x=15 to x=19, and only do the interpolaton
> based on the data from x=1 to x=15. Then I will probably do a very good job
> fitting the curve. However, I want to find a systematic method that works
> automatically and programmatically for all such curves but with varying
> turning points. It's going to be troublesome if every time I have to
> physically inspect the curves use my eyes manually.
>
> 2. Is there a way to utilize the knowledge of the asymptotic value y=1.5?
> I tried to do polynomial fit and other cubic line fit for x=1 to 19 and then
> x=100000(at which y=1.5), but the result is not very good.

you won't get a polynomial to have asymtotic behavior like that.


> There must be a way to exploit the smoothness of the curve and recover the
> ideal curve(the red one) based on the partial real data(the blue curve, up
> to x=19, before it diverges)...
>
> Please shed some lights on me! Thanks a lot!

you need to fit something else (i have one possibility below) to your
data instead of polynomials.  if you use the proir "knowledge of the
asymptotic value y=1.5" and your curve is always smaller than this
asymptote, then express it as log(1.5 - y) and fit a polynomial to
that.

   p(x[n]) = log(1.5 - y[n]) + e[n]

where

   p(x) = a0 + a1*x + a2*x^2 + a3*x^3 + ... + aM*x^M

determine a set of coefs, {a0, a1, a2, ... aM}, so that the error e[n]
is minimized by some reckoning (usually least mean square is the
easiest).

that's my quick-and-dirty approach.

r b-j

On Jul 10, 8:18 pm, "Linus Utopia" <linus_uto...@gmail.com> wrote:
> How to interpolate this curve?
>
> Hi all,
>
> Please take a look at the following plot:
>
> http://img63.imageshack.us/img63/5599/gggyf5.jpg
>
> There are two curves:
>
> The red one is the ideal one and the blue one is the real data from
> experiments.
>
> Due to some numerical instability, the real data strays away from the ideal
> curve at around x=15, and then begins oscillates weirdly after x=19. Since
> the real data is definitely wrong beyond x=19, I only capture the real data
> from x=1 to x=19.
>
> I also know that at x=+infinity, the asymptotic value of the curve should be
> 1.5. That's to say, the ideal curve slowly converges to y=1.5 from below and
> in theory it never touches y=1.5, but numerically, it can reach y=1.5 when x
> is a very large number.
>
> The questions is how to recover the ideal curve(the red one) from the
> portion of the real data(the blue curve, up to x=19). More specifically:
>
> 1. Is there a symtematic method to detect the turning point starting from
> which the real curve begins to stray away from the ideal curve (here is
> x=15)?
>
> If I throw away the data from x=15 to x=19, and only do the interpolaton
> based on the data from x=1 to x=15. Then I will probably do a very good job
> fitting the curve. However, I want to find a systematic method that works
> automatically and programmatically for all such curves but with varying
> turning points. It's going to be troublesome if every time I have to
> physically inspect the curves use my eyes manually.
>
> 2. Is there a way to utilize the knowledge of the asymptotic value y=1.5?
> I tried to do polynomial fit and other cubic line fit for x=1 to 19 and then
> x=100000(at which y=1.5), but the result is not very good.
>
> There must be a way to exploit the smoothness of the curve and recover the
> ideal curve(the red one) based on the partial real data(the blue curve, up
> to x=19, before it diverges)...
>
> Please shed some lights on me! Thanks a lot!

A single point in an unexpected place is a 'sport.'  A large number of
deviating points is a 'trend.'  I think it is important to understand
why the points do not lie on the expected graph.  If it were me, I
would put most of my energy into investigating that.  Could it be that
your ideal model does not follow the real world the way that you think
it ought to?  For instance, if it is a graph of energy from a chemical
reaction, could there be a secondary reaction taking place?

To fit to a known equation with a number of unknown constant terms,
you could use Levenberg-Marquardt.  I found this to work well:
http://sourceforge.net/project/showfiles.php?group_id=122252

On Jul 10, 11:52 pm, robert bristow-johnson
<r...@audioimagination.com> wrote:
>
> you need to fit something else (i have one possibility below) to your
> data instead of polynomials.  if you use the proir "knowledge of the
> asymptotic value y=1.5" and your curve is always smaller than this
> asymptote, then express it as log(1.5 - y) and fit a polynomial to
> that.
>
>    p(x[n]) = log(1.5 - y[n]) + e[n]

another possibility is:

     p(x[n]) = 1/(1.5 - y[n]) + e[n]

where

     p(x) = a0 + a1*x + a2*x^2 + a3*x^3 + ... + aM*x^M

again determine a set of coefs, {a0, a1, a2, ... aM}, so that the
error e[n] is minimized by some metric you like.

r b-j

If you don't have any theoretical explanation why the data should
deviate from your model curve beyond x=15, then this is likely due to
be a numerical effect caused by your data processing. For instance, if
the latter involves differences of near equal numbers, then the
numerical accuracy (i.e. the number of decimal places) your program
uses becomes crucial. Even standard double precision (16 decimal
places) might not be enough in certain cases. I have encountered
several problems in the past which forced me to use four-fold
numerical accuracy (33 decimal places) in programs, which is
unfortunately not available in many compilers as it effectively
amounts to 128 bit accuracy (although some 64 bit compilers achieve
this by patching numbers together (which was how it worked in my
case)).

So the first thing I would do is to increase numerical accuracy and
see whether this changes things.

Thomas

"Linus Utopia" <linus_utopia@gmail.com> wrote in message 
news:f71i94$gs5$1@news.Stanford.EDU...
: How to interpolate this curve?
:
: Hi all,
:
: Please take a look at the following plot:
:
: http://img63.imageshack.us/img63/5599/gggyf5.jpg
:
: There are two curves:
:
: The red one is the ideal one and the blue one is the real data from
: experiments.
:
: Due to some numerical instability, the real data strays away from the 
ideal
: curve at around x=15, and then begins oscillates weirdly after x=19. Since
: the real data is definitely wrong

HAHAHA!

The sky should be green and human beings should have five eyes
and two noses according to theory, but the real data is definitely wrong.

You need help. See a psychiatrist.

On Jul 11, 4:18 am, "Linus Utopia" <linus_uto...@gmail.com> wrote:
> How to interpolate this curve?
>
> Hi all,
>
> Please take a look at the following plot:
>
> http://img63.imageshack.us/img63/5599/gggyf5.jpg
>
> There are two curves:
>
> The red one is the ideal one and the blue one is the real data from
> experiments.
>
> Due to some numerical instability, the real data strays away from the ideal
> curve at around x=15, and then begins oscillates weirdly after x=19. Since
> the real data is definitely wrong beyond x=19, I only capture the real data
> from x=1 to x=19.
>
> I also know that at x=+infinity, the asymptotic value of the curve should be
> 1.5. That's to say, the ideal curve slowly converges to y=1.5 from below and
> in theory it never touches y=1.5, but numerically, it can reach y=1.5 when x
> is a very large number.
>
> The questions is how to recover the ideal curve(the red one) from the
> portion of the real data(the blue curve, up to x=19). More specifically:
>
> 1. Is there a symtematic method to detect the turning point starting from
> which the real curve begins to stray away from the ideal curve (here is
> x=15)?
>
> If I throw away the data from x=15 to x=19, and only do the interpolaton
> based on the data from x=1 to x=15. Then I will probably do a very good job
> fitting the curve. However, I want to find a systematic method that works
> automatically and programmatically for all such curves but with varying
> turning points. It's going to be troublesome if every time I have to
> physically inspect the curves use my eyes manually.
>
> 2. Is there a way to utilize the knowledge of the asymptotic value y=1.5?
> I tried to do polynomial fit and other cubic line fit for x=1 to 19 and then
> x=100000(at which y=1.5), but the result is not very good.
>
> There must be a way to exploit the smoothness of the curve and recover the
> ideal curve(the red one) based on the partial real data(the blue curve, up
> to x=19, before it diverges)...
>
> Please shed some lights on me! Thanks a lot!


Maple has several procedures for fitting a curve; see especially the
Statistics[Regression] procedures and the CurveFitting and
Optimization packages.

As others have pointed out, a polynomial will not give the asymptotic
behavior that you seek.  Neither will log(x), because it too goes to
infinity. Something like 1.5-P1(x)/P2(x), where P1 and P2 are
polynomials and degree(P1)<degree(P2), should work well; in that case,
try the CurveFitting:-RationalInterpolation procedure.

On Jul 11, 7:30 am, doug.kee...@informath.org wrote:
> On Jul 11, 4:18 am, "Linus Utopia" <linus_uto...@gmail.com> wrote:

>
> As others have pointed out, a polynomial will not give the asymptotic
> behavior that you seek.  Neither will log(x), because it too goes to
> infinity.

log(1.5 - f(x)) goes to infinity when x does.

also so does a polynomial of x.

> Something like 1.5-P1(x)/P2(x), where P1 and P2 are
> polynomials and degree(P1)<degree(P2), should work well;

that's another good one.  and the log() of that also goes to infinity.

r b-j

Androcles wrote:
> "Linus Utopia" wrote
> : Due to some numerical instability, the real data strays away from the ideal
> : curve at around x=15, and then begins oscillates weirdly after x=19. Since
> : the real data is definitely wrong
> 
> HAHAHA!
> 
> The sky should be green and human beings should have five eyes
> and two noses according to theory, but the real data is definitely wrong.
> 
> You need help. See a psychiatrist.

So, you've never collected data at the limit of instrument resolution, 
derived another quantity based on said data and had it go off into the 
weeds?

You need to grow up. See an adult.

On 11 Jul, 10:53, Thomas <thomas.s...@gmail.com> wrote:
> If you don't have any theoretical explanation why the data should
> deviate from your model curve beyond x=15, then this is likely due to
> be a numerical effect caused by your data processing. For instance, if
> the latter involves differences of near equal numbers, then the
> numerical accuracy (i.e. the number of decimal places) your program
> uses becomes crucial. Even standard double precision (16 decimal
> places) might not be enough in certain cases. I have encountered
> several problems in the past which forced me to use four-fold
> numerical accuracy (33 decimal places) in programs, which is
> unfortunately not available in many compilers as it effectively
> amounts to 128 bit accuracy (although some 64 bit compilers achieve
> this by patching numbers together (which was how it worked in my
> case)).
>
> So the first thing I would do is to increase numerical accuracy and
> see whether this changes things.
>
> Thomas

Just an add-on to this: if you are already using double precision in
your program and have no immediate way of improving this, you could
instead reduce the accuracy to single precision and see whether this
leads to any changes. If the accuracy is the cause of your problem,
you should see then the curve deviate already for smaller x. It may
not solve your problem, but at least you know what the cause is and
what potentially to do about it.

Thomas