Idea exploration - Image stabilization by means of software.

Basically let the camera take it's picture though user's hand is shaky, and
use a
mechanism to record down the angle of user's handshake in steps of 1us, now
I get V = [v0, v1, ...v(n-1)]; With this vector, I calculate displacement
vector
the image has moved in terms of number of pixels P.

Assume the image sensor's voltage rises linearly w.r.t exposure time (or any
charactized waveform).

I slice the image into n slices by dividing each pixel's RGB data into RGB/n
(depending on the waveform of the sensor's characters), and use a technique
similar to the motion compensation, move the slices according to
displacement
vector P and add them together to reconstruct final image.

How does my idea work?

On a sunny day (Thu, 16 Jun 2005 18:40:37 +0800) it happened "Kris Neot"
<Kris.Neot@hotmail.com> wrote in <42b15483$1@news.starhub.net.sg>:
These systems (software stabilization) are already in many consumer video
cameras.
There are also stand alone (MS windows) programs that will stabilize an
existing shaking recording.


>Basically let the camera take it's picture though user's hand is shaky, and
>use a
>mechanism to record down the angle of user's handshake in steps of 1us, now
You will at best get a set of motion vectors per frame, so 1us makes no sense?
Think 40mS (@25fps).

>I get V = [v0, v1, ...v(n-1)]; With this vector, I calculate displacement
>vector
>the image has moved in terms of number of pixels P.
And of cause mpeg2 encoding already gives us these vectors for free.

>Assume the image sensor's voltage rises linearly w.r.t exposure time (or any
>charactized waveform).
I do not see what motion in one direction or an other has to do with exposure
time.
Exposure time has to do with how much sensitivity you have to light.
Of cause when your exposure time get freaky long, you get motion blur.


>I slice the image into n slices by dividing each pixel's RGB data into RGB/n
>(depending on the waveform of the sensor's characters), and use a technique
>similar to the motion compensation, move the slices according to
>displacement
>vector P and add them together to reconstruct final image.
I have lost you here.


>How does my idea work?
What idea?

[X-post, F'up2 cut down by me, should have been done by you!]

In comp.arch.embedded Kris Neot <Kris.Neot@hotmail.com> wrote:

> Basically let the camera take it's picture though user's hand is
> shaky, and use a mechanism to record down the angle of user's
> handshake in steps of 1us, 

1 microsecond won't do you any good.  No human can move anything at a
frequency of 1 MHz.  

Anyway, what happens during the exposition for a single frame will be
completely impossible to correct for after the fact.  That's why some
current consumer cameras do this in a different way: they use movable
parts in the optical system to compensate shaking *before* any light
hits the sensors.  That's the only way it becomes possible to do
in-frame shake compensation.

> now I get V = [v0, v1, ...v(n-1)]; With this vector, I calculate
> displacement vector the image has moved in terms of number of pixels
> P.

Displacement alone is also useless.  You also have to take care of
rotation --- that's actually more important than displacement.

-- 
Hans-Bernhard Broeker (broeker@physik.rwth-aachen.de)
Even if all the snow were burnt, ashes would remain.

Jan Panteltje wrote:
> <snip>
> 
>>Assume the image sensor's voltage rises linearly w.r.t exposure time (or any
>>charactized waveform).
> 
> I do not see what motion in one direction or an other has to do with exposure
> time.
> Exposure time has to do with how much sensitivity you have to light.
> Of cause when your exposure time get freaky long, you get motion blur.
> 
> <snip>
> 
>>How does my idea work?
> 
> What idea?
> 

I believe he's talking about a single frame exposure not video.  He's 
talking about motion during the exposure and I assume a problem of 
varying affects of the motion on the different sensors (e.g. R,G and B) 
that combine to give the final image.

I don't have any feedback for him other than to suggest he try to state 
his problem better next time especially when cross-posting so broadly.

-- ced



-- 
Chuck Dillon
Senior Software Engineer
NimbleGen Systems Inc.

On Thu, 16 Jun 2005 08:58:16 -0500, Chuck Dillon <spam@nimblegen.com>
wrote:

>I believe he's talking about a single frame exposure not video.  He's 
>talking about motion during the exposure and I assume a problem of 
>varying affects of the motion on the different sensors (e.g. R,G and B) 
>that combine to give the final image.

If the end product is a single frame, what prevents from taking
multiple exposures and combine the shifted exposures into a single
frame ?

In ground based optical astronomy, the telescope is stable, but the
atmospheric turbulence deflect the light rays quite frequently to
slightly different directions. Taking a long exposure would cause
quite blurred pictures. Take multiple short exposures, shift each
exposure according to the atmospheric deflection and combine the
exposures. Look for "stacking" and "speckle-interferometer".

To do the image stabilisation in a camera, the MPEG motion vectors
could be used to determine the global movement between exposures, but
unfortunately getting reliable motion vectors from low SNR exposures
is quite challenging. Of course if 3 axis velocity sensors are
available, these can be used to get rid of any ambiguous motion
vectors and also simplify searching for motion vectors. 

With reliable motion vectors for these multiple low SNR exposures, the
exposures can be shifted accordingly, before summing the exposures to
a single high SNR frame.

Paul
 

Correct, my idea is much liks multiple short time exposure then
combine them while taking into account of the displacement. If
similar techniques have been done in the telescopes, then it is
a baked bean. No need to explore it further.



"Paul Keinanen" <keinanen@sci.fi> wrote in message
news:sjh3b15dd6l1ppd9i4oilql87nlfis7mpo@4ax.com...
> On Thu, 16 Jun 2005 08:58:16 -0500, Chuck Dillon <spam@nimblegen.com>
> wrote:
>
> >I believe he's talking about a single frame exposure not video.  He's
> >talking about motion during the exposure and I assume a problem of
> >varying affects of the motion on the different sensors (e.g. R,G and B)
> >that combine to give the final image.
>
> If the end product is a single frame, what prevents from taking
> multiple exposures and combine the shifted exposures into a single
> frame ?
>
> In ground based optical astronomy, the telescope is stable, but the
> atmospheric turbulence deflect the light rays quite frequently to
> slightly different directions. Taking a long exposure would cause
> quite blurred pictures. Take multiple short exposures, shift each
> exposure according to the atmospheric deflection and combine the
> exposures. Look for "stacking" and "speckle-interferometer".
>
> To do the image stabilisation in a camera, the MPEG motion vectors
> could be used to determine the global movement between exposures, but
> unfortunately getting reliable motion vectors from low SNR exposures
> is quite challenging. Of course if 3 axis velocity sensors are
> available, these can be used to get rid of any ambiguous motion
> vectors and also simplify searching for motion vectors.
>
> With reliable motion vectors for these multiple low SNR exposures, the
> exposures can be shifted accordingly, before summing the exposures to
> a single high SNR frame.
>
> Paul
>

Found a flaw in my idea after a little test.

My deer is a perfectly exposed picture and my trick is applied to it,
I ended up with a very dull pic, thus, I believe, with a picture that
has motion blur, the outcome will be even more worse after application
of my idea. To improve my idea, I need the image sensor to run at
1000fps, which is impossible.

a = imread('deer.jpg'); % size = (864,576,3)
b = int32(a);
c = zeros(880, 585, 3);
cz = zeros(880, 585, 3);
d = [5,3; 8,3; 2,5; 3,7; 5,1; 2,6; 4,4; 2,7];

for i = 1:1:8
    t = cz;
    t(d(i,1):(d(i,1)+864-1), d(i,2):(d(i,2)+576-1), 1:3) = a;
    c = c + t;
    fprintf('done with %d\n', i);
end
c = c / 8;
c = uint8(c);

figure(1);
image(a);
figure(2);
image(c);





"Kris Neot" <Kris.Neot@hotmail.com> wrote in message
news:42b15483$1@news.starhub.net.sg...
> Basically let the camera take it's picture though user's hand is shaky,
and
> use a
> mechanism to record down the angle of user's handshake in steps of 1us,
now
> I get V = [v0, v1, ...v(n-1)]; With this vector, I calculate displacement
> vector
> the image has moved in terms of number of pixels P.
>
> Assume the image sensor's voltage rises linearly w.r.t exposure time (or
any
> charactized waveform).
>
> I slice the image into n slices by dividing each pixel's RGB data into
RGB/n
> (depending on the waveform of the sensor's characters), and use a
technique
> similar to the motion compensation, move the slices according to
> displacement
> vector P and add them together to reconstruct final image.
>
> How does my idea work?
>
>
>

Are you taking pictures of stationary things from a stationary camera?  That's
what the astronomy case involves.  But since you are concerned with a moving
camera, I assumed you were talking about pictures taken with a hand-held camera.
In that case, the reason the shake creates blur is that the exposure is too
long.  And the reason for the long exposure is lack of light (due to lens
limitations, ambient light conditions, or both).  If you were to shorted the
exposure, you would end up with a picture that is too dark.

"Kris Neot" <Kris.Neot@hotmail.com> wrote in message
news:42b22b58@news.starhub.net.sg...
> Correct, my idea is much liks multiple short time exposure then
> combine them while taking into account of the displacement. If
> similar techniques have been done in the telescopes, then it is
> a baked bean. No need to explore it further.
>
>
>
> "Paul Keinanen" <keinanen@sci.fi> wrote in message
> news:sjh3b15dd6l1ppd9i4oilql87nlfis7mpo@4ax.com...
> > On Thu, 16 Jun 2005 08:58:16 -0500, Chuck Dillon <spam@nimblegen.com>
> > wrote:
> >
> > >I believe he's talking about a single frame exposure not video.  He's
> > >talking about motion during the exposure and I assume a problem of
> > >varying affects of the motion on the different sensors (e.g. R,G and B)
> > >that combine to give the final image.
> >
> > If the end product is a single frame, what prevents from taking
> > multiple exposures and combine the shifted exposures into a single
> > frame ?
> >
> > In ground based optical astronomy, the telescope is stable, but the
> > atmospheric turbulence deflect the light rays quite frequently to
> > slightly different directions. Taking a long exposure would cause
> > quite blurred pictures. Take multiple short exposures, shift each
> > exposure according to the atmospheric deflection and combine the
> > exposures. Look for "stacking" and "speckle-interferometer".
> >
> > To do the image stabilisation in a camera, the MPEG motion vectors
> > could be used to determine the global movement between exposures, but
> > unfortunately getting reliable motion vectors from low SNR exposures
> > is quite challenging. Of course if 3 axis velocity sensors are
> > available, these can be used to get rid of any ambiguous motion
> > vectors and also simplify searching for motion vectors.
> >
> > With reliable motion vectors for these multiple low SNR exposures, the
> > exposures can be shifted accordingly, before summing the exposures to
> > a single high SNR frame.
> >
> > Paul

"Jon Harris" <jon_harrisTIGER@hotmail.com> wrote in message
news:11b4mltr031dld2@corp.supernews.com...
> Are you taking pictures of stationary things from a stationary camera?
That's
> what the astronomy case involves.  But since you are concerned with a
moving
> camera, I assumed you were talking about pictures taken with a hand-held
camera.
> In that case, the reason the shake creates blur is that the exposure is
too
> long.  And the reason for the long exposure is lack of light (due to lens
> limitations, ambient light conditions, or both).  If you were to shorted
the
> exposure, you would end up with a picture that is too dark.
>

Yeah, now I understand. My idea is flawed.

"Jon Harris" <jon_harrisTIGER@hotmail.com> writes:

> Are you taking pictures of stationary things from a stationary camera?
> That's what the astronomy case involves.  But since you are concerned
> with a moving camera, I assumed you were talking about pictures taken
> with a hand-held camera.  In that case, the reason the shake creates
> blur is that the exposure is too long.  And the reason for the long
> exposure is lack of light (due to lens limitations, ambient light
> conditions, or both).  If you were to shorted the exposure, you would
> end up with a picture that is too dark.

So - comp.dspers - does anybody know what algorithms hand-held cameras
and hand-held video recorders use to reduce camera shake?  Surely some
estimation of movement and compensation is possible in the same way as
astronomy?  But does your average digital camera have enough CPU?  So
what is used in practice - anything at all?


Tony