Idea exploration - Image stabilization by means of software.

I suspect Canon does it by moving the lens a bit to compensate the movement.
Canon lens with IS has a mechanical part that trembles when you shoot a
picture.



<tony.nospam@nospam.tonyRobinson.com> wrote in message
news:87r7f1qvnu.fsf@tonyRobinson.com...
> "Jon Harris" <jon_harrisTIGER@hotmail.com> writes:
>
> > Are you taking pictures of stationary things from a stationary camera?
> > That's what the astronomy case involves.  But since you are concerned
> > with a moving camera, I assumed you were talking about pictures taken
> > with a hand-held camera.  In that case, the reason the shake creates
> > blur is that the exposure is too long.  And the reason for the long
> > exposure is lack of light (due to lens limitations, ambient light
> > conditions, or both).  If you were to shorted the exposure, you would
> > end up with a picture that is too dark.
>
> So - comp.dspers - does anybody know what algorithms hand-held cameras
> and hand-held video recorders use to reduce camera shake?  Surely some
> estimation of movement and compensation is possible in the same way as
> astronomy?  But does your average digital camera have enough CPU?  So
> what is used in practice - anything at all?
>
>
> Tony
>

On Thu, 16 Jun 2005 22:08:13 -0700, "Jon Harris"
<jon_harrisTIGER@hotmail.com> wrote:

>Are you taking pictures of stationary things from a stationary camera?  That's
>what the astronomy case involves.  But since you are concerned with a moving
>camera, I assumed you were talking about pictures taken with a hand-held camera.
>In that case, the reason the shake creates blur is that the exposure is too
>long.  And the reason for the long exposure is lack of light (due to lens
>limitations, ambient light conditions, or both).  If you were to shorted the
>exposure, you would end up with a picture that is too dark.
>

[Stuff snipped]

I have read a bit about the astronomical image processing mention in
one of the other posts. One thing that came up quite often, is that
CCD sensors are linear. i.e. double the exposure time gives double the
energy recieved. (They do not mention CMOS, but it is probably the
same.)
I agree that blurring occurs when taking a single photo when the
camaera is moved and the exposure is to long. If one can say take in
stead of 1x 2sec exposure, take 20x 0.1 sec exposures, and stack them
into a single picture in software, one should end up close to the
exposure level of the 2sec picture, but without the blur.
In the astronomical case, the "short" exposure pictures are in minutes
for a total exposure time of hours. Of course the subject must not
move during the "short" exposure time. 
If one scaled this down so that the "short" exposure time is in
milli-seconds or even micro-seconds depending on what the sensors can
do, then one can do the same thing just at a much faster scale.
What is the minimum exposure time for a current CMOS sensor before one
just see the inherant sensor noise ?

Regards
   Anton Erasmu

On Thu, 16 Jun 2005 22:08:13 -0700, "Jon Harris"
<jon_harrisTIGER@hotmail.com> wrote:

>Are you taking pictures of stationary things from a stationary camera?  That's
>what the astronomy case involves.  

Have you ever wondered why stars twinkle or looked at the Moon with a
large magnification (100x or more) ?

While the target and the telescope are both stationary (or actually
moving in a predictable way) the target appears to be bouncing around
a lot due to the atmospheric turbulence. The effect would be similar
as if the telescope was shaking. Thus, the tricks used in astronomy
should be applicable also for photographic camera stabilisation.

Paul

Kris Neot wrote:

> Correct, my idea is much liks multiple short time exposure then
> combine them while taking into account of the displacement. If
> similar techniques have been done in the telescopes, then it is
> a baked bean. No need to explore it further.

Amateur astromoners are now doing it routinely with modest hardware to 
get stunning results from "lucky" exposures. Do a search on "registax" 
and Toucam and something like QUAICG (I forget the spelling).

http://members.dialstar.com/asrc/T_Davis6.htm

Correlate, stack and add the highest contrast images and bin the dross 
works amazingly well if you have enough signal to noise.

Regards,
Martin Brown

> "Paul Keinanen" <keinanen@sci.fi> wrote in message
> news:sjh3b15dd6l1ppd9i4oilql87nlfis7mpo@4ax.com...
> 
>>On Thu, 16 Jun 2005 08:58:16 -0500, Chuck Dillon <spam@nimblegen.com>
>>wrote:
>>
>>
>>>I believe he's talking about a single frame exposure not video.  He's
>>>talking about motion during the exposure and I assume a problem of
>>>varying affects of the motion on the different sensors (e.g. R,G and B)
>>>that combine to give the final image.
>>
>>If the end product is a single frame, what prevents from taking
>>multiple exposures and combine the shifted exposures into a single
>>frame ?
>>
>>In ground based optical astronomy, the telescope is stable, but the
>>atmospheric turbulence deflect the light rays quite frequently to
>>slightly different directions. Taking a long exposure would cause
>>quite blurred pictures. Take multiple short exposures, shift each
>>exposure according to the atmospheric deflection and combine the
>>exposures. Look for "stacking" and "speckle-interferometer".
>>
>>To do the image stabilisation in a camera, the MPEG motion vectors
>>could be used to determine the global movement between exposures, but
>>unfortunately getting reliable motion vectors from low SNR exposures
>>is quite challenging. Of course if 3 axis velocity sensors are
>>available, these can be used to get rid of any ambiguous motion
>>vectors and also simplify searching for motion vectors.
>>
>>With reliable motion vectors for these multiple low SNR exposures, the
>>exposures can be shifted accordingly, before summing the exposures to
>>a single high SNR frame.
>>
>>Paul

You don't need to record the motion vectors. They can be derived after 
the event by cross correlating the images.

Kris Neot wrote:
> Found a flaw in my idea after a little test.
>
> My deer is a perfectly exposed picture and my trick is applied to it,
> I ended up with a very dull pic, thus, I believe, with a picture that
> has motion blur, the outcome will be even more worse after application
> of my idea. To improve my idea, I need the image sensor to run at
> 1000fps, which is impossible.
Well, if you are just filming a swinging ball, have a look at this:
http://www.2d3.com/jsp/products/product-examples.jsp?product=11
download the dolphin movie.
Not sure is any help, but the result (improvement) is spectacular.

Anton Erasmus wrote:
> <jon_harrisTIGER@hotmail.com> wrote:
> 
>> Are you taking pictures of stationary things from a stationary
>> camera?  That's what the astronomy case involves.  But since you
>> are concerned with a moving camera, I assumed you were talking
>> about pictures taken with a hand-held camera. In that case, the
>> reason the shake creates blur is that the exposure is too long. 
>> And the reason for the long exposure is lack of light (due to
>> lens limitations, ambient light conditions, or both).  If you
>> were to shorted the exposure, you would end up with a picture
>> that is too dark.
> 
> [Stuff snipped]
> 
> I have read a bit about the astronomical image processing mention
> in one of the other posts. One thing that came up quite often, is
> that CCD sensors are linear. i.e. double the exposure time gives
> double the energy recieved. (They do not mention CMOS, but it is
> probably the same.)
>
> I agree that blurring occurs when taking a single photo when the
> camaera is moved and the exposure is to long. If one can say take
> in stead of 1x 2sec exposure, take 20x 0.1 sec exposures, and
> stack them into a single picture in software, one should end up
> close to the exposure level of the 2sec picture, but without the
> blur. In the astronomical case, the "short" exposure pictures are
> in minutes for a total exposure time of hours. Of course the
> subject must not move during the "short" exposure time.
>
> If one scaled this down so that the "short" exposure time is in
> milli-seconds or even micro-seconds depending on what the sensors
> can do, then one can do the same thing just at a much faster
> scale. What is the minimum exposure time for a current CMOS sensor
> before one just see the inherant sensor noise ?

I think, at any particular time, you have an accumulated picture
and a picture increment to combine.  Around any particular point
you have a matrix of adjacent points, described by the indices -1,
0, +1 in each direction.  To do the combination, and arrive at a
new accumulated picture, you 'just' have to form the 9 value matrix
and combine all points accordingly.  Now you can get a new picture
increment, and recompute the matrix.  You combine the old into the
new, so you never have to worry about more than one increment of
motion.

Forming the matrix will depend on a herd of heuristics, such as
'where is the CG of the input points', 'where is the minimum',
'where is the maximum'.

This is pure speculation on my part.  I have no knowledge of real
world algorithms here.  It's how I would go about it.  I might want
a buffer for picture increment, so that the combining can take
place in parallel with the input of the next increment.

-- 
Chuck F (cbfalconer@yahoo.com) (cbfalconer@worldnet.att.net)
   Available for consulting/temporary embedded and systems.
   <http://cbfalconer.home.att.net>  USE worldnet address!

<tony.nospam@nospam.tonyRobinson.com> wrote in message
news:87r7f1qvnu.fsf@tonyRobinson.com...
> "Jon Harris" <jon_harrisTIGER@hotmail.com> writes:
>
> > Are you taking pictures of stationary things from a stationary camera?
> > That's what the astronomy case involves.  But since you are concerned
> > with a moving camera, I assumed you were talking about pictures taken
> > with a hand-held camera.  In that case, the reason the shake creates
> > blur is that the exposure is too long.  And the reason for the long
> > exposure is lack of light (due to lens limitations, ambient light
> > conditions, or both).  If you were to shorted the exposure, you would
> > end up with a picture that is too dark.
>
> So - comp.dspers - does anybody know what algorithms hand-held cameras
> and hand-held video recorders use to reduce camera shake?  Surely some
> estimation of movement and compensation is possible in the same way as
> astronomy?  But does your average digital camera have enough CPU?  So
> what is used in practice - anything at all?

As explained in a previous past, using electronic compensation for single-frame
images (i.e. with still cameras) is not useful.  In that case, either an optical
element or the sensor itself is moved to compensate for the camera movement.
This is usually done with a servo loop of some kind.
For video cameras, the same "optical image stabilization" just described is used
in some higher-end camcorders, but most of the consumer camcorders do this
digitally after capturing the image.  Presumably, this is accomplished by
looking at the frame-to-frame differences and essentially cropping different
areas from each frame in order to keep the video as stable as possible.  I don't
know much more about the algorithms than that.

"Anton Erasmus" <nobody@spam.prevent.net> wrote in message 
news:1119000420.d54828b53b9bcd51f76b2b5b640103a6@teranews...
>>
> I have read a bit about the astronomical image processing mention in
> one of the other posts. One thing that came up quite often, is that
> CCD sensors are linear. i.e. double the exposure time gives double the
> energy recieved. (They do not mention CMOS, but it is probably the
> same.)
> I agree that blurring occurs when taking a single photo when the
> camaera is moved and the exposure is to long. If one can say take in
> stead of 1x 2sec exposure, take 20x 0.1 sec exposures, and stack them
> into a single picture in software, one should end up close to the
> exposure level of the 2sec picture, but without the blur.
> In the astronomical case, the "short" exposure pictures are in minutes
> for a total exposure time of hours. Of course the subject must not
> move during the "short" exposure time.
> If one scaled this down so that the "short" exposure time is in
> milli-seconds or even micro-seconds depending on what the sensors can
> do, then one can do the same thing just at a much faster scale.
> What is the minimum exposure time for a current CMOS sensor before one
> just see the inherant sensor noise ?

Actually in speckle interferometry, around 30 exposures are done per second. 
The idea is the atmosphere is comprised of cells (roughly 10 to 20 cm in 
diameter) that are stationary during a short time frame like a 30th of a 
second. If a telescope's objective is smaller than 10 cm in diameter, you 
won't have a speckle problem.  A sequence of pictures taken at a high rate 
are overlapped so the noise tends to cancel out. A large study of binary 
star orbits has been carried out this way. Looking at point sources (stars) 
allows this simple speckle trick to actually work. With extended images, it 
becomes a whole different problem.

Both Canon and Nikon perform Image Stabilization (Canon's name) and 
Vibration Reduction (Nikon's name) by actually rotating an optical element 
(prism) located at or near an optical center of the lens. Translating the 
sensor requires more effort that rotating a prism to effectively remove 
motion. And yes the lens employs inertial sensors to effect motion 
reduction. I can say with personal experience that they work extremely well. 
Most tests with such optical systems yield a two to three F stop increase in 
exposure time for the same amount of blurring without compensation.

Clay Turner

> "Anton Erasmus" <nobody@spam.prevent.net> wrote in message
> news:1119000420.d54828b53b9bcd51f76b2b5b640103a6@teranews...
> >>
> > I have read a bit about the astronomical image processing mention in
> > one of the other posts. One thing that came up quite often, is that
> > CCD sensors are linear. i.e. double the exposure time gives double the
> > energy recieved. (They do not mention CMOS, but it is probably the
> > same.)
> > I agree that blurring occurs when taking a single photo when the
> > camaera is moved and the exposure is to long. If one can say take in
> > stead of 1x 2sec exposure, take 20x 0.1 sec exposures, and stack them
> > into a single picture in software, one should end up close to the
> > exposure level of the 2sec picture, but without the blur.

I'm assuming that the software wouldn't simply stack the pictures exactly on top
of each other, but move each one around slightly so as to best align them.

One potential difficulty is the time it takes to read out the data from the
sensor and "clear" it for the next exposure could be significant, especially for
very short exposures.  I don't know specifics, but for example, it might take
3-4 seconds to take your 2 seconds worth of exposure.  Clearly that is not a
good thing!

> > In the astronomical case, the "short" exposure pictures are in minutes
> > for a total exposure time of hours. Of course the subject must not
> > move during the "short" exposure time.

Another difficulty I see is that when you have a severely underexposed picture,
you are going to lose a lot of shadow detail since it ends up below the noise
floor .  A perfectly linear sensor with no noise wouldn't have this problem, but
real-world ones certainly do.  Consider, for example, a picture that has a
black-to-white gradient.  When exposed normally, the camera records levels from
0 (full black) to maximum (full white).  But with an exposure that is 1/20 of
the proper value, much of the dark gray portion will be rendered as 0 (full
black). Even after you add the 20 exposures together, you still get full black
for those dark gray portions.  So everything below a certain dark gray level is
clipped to full black.

In astro-photography, this might not be such a problem, as the unlit sky is
supposed to be black anyway, but with typical photographic scenes, it would be.

> > If one scaled this down so that the "short" exposure time is in
> > milli-seconds or even micro-seconds depending on what the sensors can
> > do, then one can do the same thing just at a much faster scale.
> > What is the minimum exposure time for a current CMOS sensor before one
> > just see the inherant sensor noise ?

As mentioned above, the shorter you try to make your individual exposures, the
more the read/clear time becomes a factor.  State-of-the-art high mega-pixel
cameras today can shoot maybe 8-10 frames/second.  So I doubt the sensors are
capable with dealing with micro-seconds or even low milli-seconds.  Video
cameras can of course shoot 30 frames/second, but their sensors are typically
quite low resolution (i.e. <1 mega-pixel).

On Sun, 19 Jun 2005 00:57:12 -0700, "Jon Harris"
<jon_harrisTIGER@hotmail.com> wrote:

>> "Anton Erasmus" <nobody@spam.prevent.net> wrote in message
>> news:1119000420.d54828b53b9bcd51f76b2b5b640103a6@teranews...
>> >>
>> > I have read a bit about the astronomical image processing mention in
>> > one of the other posts. One thing that came up quite often, is that
>> > CCD sensors are linear. i.e. double the exposure time gives double the
>> > energy recieved. (They do not mention CMOS, but it is probably the
>> > same.)
>> > I agree that blurring occurs when taking a single photo when the
>> > camaera is moved and the exposure is to long. If one can say take in
>> > stead of 1x 2sec exposure, take 20x 0.1 sec exposures, and stack them
>> > into a single picture in software, one should end up close to the
>> > exposure level of the 2sec picture, but without the blur.
>
>I'm assuming that the software wouldn't simply stack the pictures exactly on top
>of each other, but move each one around slightly so as to best align them.
>
>One potential difficulty is the time it takes to read out the data from the
>sensor and "clear" it for the next exposure could be significant, especially for
>very short exposures.  I don't know specifics, but for example, it might take
>3-4 seconds to take your 2 seconds worth of exposure.  Clearly that is not a
>good thing!

Lots of cheap digital cameras can take short mpeg movies, so it cannot
be that slow for the sensor to recover.
If someone has better information regarding the technical specs of
these devices, as well as what the theoretical limits are, it would be
quite nice to know.

>> > In the astronomical case, the "short" exposure pictures are in minutes
>> > for a total exposure time of hours. Of course the subject must not
>> > move during the "short" exposure time.
>
>Another difficulty I see is that when you have a severely underexposed picture,
>you are going to lose a lot of shadow detail since it ends up below the noise
>floor .  A perfectly linear sensor with no noise wouldn't have this problem, but
>real-world ones certainly do.  Consider, for example, a picture that has a
>black-to-white gradient.  When exposed normally, the camera records levels from
>0 (full black) to maximum (full white).  But with an exposure that is 1/20 of
>the proper value, much of the dark gray portion will be rendered as 0 (full
>black). Even after you add the 20 exposures together, you still get full black
>for those dark gray portions.  So everything below a certain dark gray level is
>clipped to full black.

Yes one would need a sensor with a low noise floor. The lower the
better. AFAIK the more pixels they pack into a sensor, the higher the
noise floor. Currently the whole emphasis is on producing sensors with
more pixels. If the emphasis was on producing sensors with very low
noise floor, I am sure a suitable sensor can be developed.

>In astro-photography, this might not be such a problem, as the unlit sky is
>supposed to be black anyway, but with typical photographic scenes, it would be.
>
>> > If one scaled this down so that the "short" exposure time is in
>> > milli-seconds or even micro-seconds depending on what the sensors can
>> > do, then one can do the same thing just at a much faster scale.
>> > What is the minimum exposure time for a current CMOS sensor before one
>> > just see the inherant sensor noise ?
>
>As mentioned above, the shorter you try to make your individual exposures, the
>more the read/clear time becomes a factor.  State-of-the-art high mega-pixel
>cameras today can shoot maybe 8-10 frames/second.  So I doubt the sensors are
>capable with dealing with micro-seconds or even low milli-seconds.  Video
>cameras can of course shoot 30 frames/second, but their sensors are typically
>quite low resolution (i.e. <1 mega-pixel).
>

Yes, but at what exposure time does one currently hit the wall, and
where does the actual theoretical wall lie ? Also if one could cool
down the sensor, by how much would the noise floor be reduced ?

Regards
  Anton Erasmus