Deblocking filter for JPEG/digital video

G'day folks,

I have come across a deblocking algorithm for JPEG (and similar) images
that seems to do a pretty good job, but I have not been able to fully
tell whether it stacks up well against the existing solutions. I was
hoping a few of you folks might give me a general thumbs up or thumbs
down before I invest more time and effort in it. Please be brutally
honest.

Visually, it seems to do a better job than the solutions I've come
across (some more than others), but I don't have any quantitative test
programs for this sort of comparison. (Other than my eyes.)

In terms of speed, I am completely in the dark, because I can't find
any speed specs on existing solutions. I know this is relatively
unimportant for JPEG, but of course is crucial for digital video
applications.

I've uploaded a simple Windows application using the filter to

http://www.assassinationscience.com/johncostella/unblock

which also includes some "before and after" images on an attached page.
(Sorry about the web page being technically feeble; there are reasons
for this.)

As far as speed is concerned, the sample app is not indicative, because
the Windows wrapper that I used was a pretty inefficient one. However,
these are the execution times that I obtained for typical images on a
1.6 GHz Pentium M:

Color 720 x 480: 66 ms
Color 640 x 480: 59 ms
Color 320 x 240: 14 ms

Grayscale 720 x 480: 31 ms
Grayscale 640 x 480: 28 ms
Grayscale 320 x 240: 6.5 ms

I'd be extremely grateful for any words of wisdom from a few of the
experts out there.

Many thanks in advance,
John Costella

=====================================
John P. Costella, PhD
Melbourne, Australia
jpcostella@hotmail.com 
assassinationscience.com/johncostella

Sorry, I thought I should make a clarification. The execution times I
gave might be a little misleading. They assumed that color images would
be fed to the algorithm in RGB format, and the results fed back in RGB
format, so they include not just the deblocking filter proper, but also
two colorspace conversions with chrominance upsampling, which together
account for around one-third of the execution times quoted.

If the filter is placed into an existing YCbCr datastream, with
sensible upsampling of CbCr already handled further downstream in the
conversion back to RGB, then the execution times on the 1.6 GHz Pentium
M are modified to:

720 x 480: 45 ms
640 x 480: 40 ms
320 x 240: 9.5 ms

Sorry for any confusion.
John Costella

=====================================
John P. Costella, PhD
Melbourne, Australia
jpcostella@hotmail.com 
assassinationscience.com/johncostella

John Costella wrote:
> G'day folks,
>
> I have come across a deblocking algorithm for JPEG (and similar) images
> that seems to do a pretty good job, but I have not been able to fully
> tell whether it stacks up well against the existing solutions. I was
> hoping a few of you folks might give me a general thumbs up or thumbs
> down before I invest more time and effort in it. Please be brutally
> honest.
>
> Visually, it seems to do a better job than the solutions I've come
> across (some more than others), but I don't have any quantitative test
> programs for this sort of comparison. (Other than my eyes.)
>
> In terms of speed, I am completely in the dark, because I can't find
> any speed specs on existing solutions. I know this is relatively
> unimportant for JPEG, but of course is crucial for digital video
> applications.
>
> I've uploaded a simple Windows application using the filter to
>
> http://www.assassinationscience.com/johncostella/unblock
>
> which also includes some "before and after" images on an attached page.
> (Sorry about the web page being technically feeble; there are reasons
> for this.)
>
> As far as speed is concerned, the sample app is not indicative, because
> the Windows wrapper that I used was a pretty inefficient one. However,
> these are the execution times that I obtained for typical images on a
> 1.6 GHz Pentium M:
>
> Color 720 x 480: 66 ms
> Color 640 x 480: 59 ms
> Color 320 x 240: 14 ms
>
> Grayscale 720 x 480: 31 ms
> Grayscale 640 x 480: 28 ms
> Grayscale 320 x 240: 6.5 ms
>
> I'd be extremely grateful for any words of wisdom from a few of the
> experts out there.

I am no expert on this subject... but the results surely look
impressive to me.

Sachin Garg [India]
http://www.sachingarg.com

Thank you, Sachin. Hopefully I will find some experts soon. ;)

The best existing results I could find on the net were from Pegasus
Imaging, which also deblocked without needing user input or fine
tuning, but their decompressor seemed to leave in some strange blocks
and other artifacts, and didn't seem to upsample the chrominance very
sensibly. However, their software goes back a decade or so, so I don't
know if it has been superseded or abandoned by now.

John

John Costella wrote:

> The best existing results I could find on the net were from Pegasus
> Imaging, which also deblocked without needing user input or fine
> tuning, but their decompressor seemed to leave in some strange blocks
> and other artifacts, and didn't seem to upsample the chrominance very
> sensibly. However, their software goes back a decade or so, so I don't
> know if it has been superseded or abandoned by now.

There's so many methods, but you'd have to implement them to benchmark 
against them ... reproducible research isn't big in image processing.

You could try benchmarking against this :
http://www.utdallas.edu/~aria/mcl/post/

Incredibly slow, but looks pretty good and the source code is available.

John Costella wrote:
> G'day folks,
>
> I have come across a deblocking algorithm for JPEG (and similar) images
> that seems to do a pretty good job, but I have not been able to fully
> tell whether it stacks up well against the existing solutions. I was
> hoping a few of you folks might give me a general thumbs up or thumbs
> down before I invest more time and effort in it. Please be brutally
> honest.
>
> Visually, it seems to do a better job than the solutions I've come
> across (some more than others), but I don't have any quantitative test
> programs for this sort of comparison. (Other than my eyes.)
>
> In terms of speed, I am completely in the dark, because I can't find
> any speed specs on existing solutions. I know this is relatively
> unimportant for JPEG, but of course is crucial for digital video
> applications.
>
> I've uploaded a simple Windows application using the filter to
>
> http://www.assassinationscience.com/johncostella/unblock
>
> which also includes some "before and after" images on an attached page.
> (Sorry about the web page being technically feeble; there are reasons
> for this.)
>
> As far as speed is concerned, the sample app is not indicative, because
> the Windows wrapper that I used was a pretty inefficient one. However,
> these are the execution times that I obtained for typical images on a
> 1.6 GHz Pentium M:
>
> Color 720 x 480: 66 ms
> Color 640 x 480: 59 ms
> Color 320 x 240: 14 ms
>
> Grayscale 720 x 480: 31 ms
> Grayscale 640 x 480: 28 ms
> Grayscale 320 x 240: 6.5 ms
>
> I'd be extremely grateful for any words of wisdom from a few of the
> experts out there.
>
> Many thanks in advance,
> John Costella
>
> =====================================
> John P. Costella, PhD
> Melbourne, Australia
> jpcostella@hotmail.com
> assassinationscience.com/johncostella

Hi, John. I have done some research on deblocking algorithms, and your
results look comparable to others that I have seen and implemented. As
you probably know, there are a multiplicity of algorithms out there,
and as of a couple years ago there were no objective tests for quality
that showed any significant benefit over SNR, which itself is known not
to be a good predictor of subjective image quality. Consequently, it
seems that every author claims his/her method is the best, but there's
really no objective test. My suspicion is that different algorithms
work better on different types of imagery.

As far as your results, I notice that blockiness was reduced, but the
ringing is still pretty noticeable and there are some mosaic patterns
(e.g. on the fence post on the left-center of the image). For
reference, here's a list of non-blocky artifacts:

http://www.wardlabs.com/technology.htm#digital

Cheers! --M

Hi,

Yes, Pegasus is still around and still the most robust and fastest (and 
protected from Forgent) JPEG available.  Our JPEG is much enhanced and 
optimized since your reference, of course.  Since you looked, we've added an 
option for chrominance smoothing whose lacking you evidently noticed.  By 
the way, our deblocking at the time was the first and it's still, as then, 
very effective for smooth color gradations in heavily compressed images. 
Pegasus deblocking works without blurring or distortion, and is a completely 
different animal from the 'cross block smoothing' referenced in the IJG 
code.

Besides being first with deblocking, we were also the first with DCT-space 
rotation, DCT-space cropping, and DCT-space brightness/contrast, and 
DCT-space requantizing also.  We didn't patent any of these, unfortunately 
for us, but great for everyone else -- it was a different world then.  We do 
have a patent application pending on additional deblocking enhancements that 
are already shipping in our latest code.  And we're still the fastest JPEG 
on the latest cpu's as well as on older cpu's.

Our deblocking is in all of our JPEG C/C++, COM and .NET toolkits, as well 
as our JPEG Wizard(tm) product if you want to test it out.  The speed hit is 
7-10% on decompression, so for speed trial (or not very compressed JPEGs), 
it should be turned off.

Hope this helps,
jack
-- 
Pegasus - BETTER DIGITAL IMAGING!
http://www.pegasusimaging.com/

"John Costella" <jpcostella@hotmail.com> wrote in message 
news:1138067495.312176.155490@f14g2000cwb.googlegroups.com...
> Thank you, Sachin. Hopefully I will find some experts soon. ;)
>
> The best existing results I could find on the net were from Pegasus
> Imaging, which also deblocked without needing user input or fine
> tuning, but their decompressor seemed to leave in some strange blocks
> and other artifacts, and didn't seem to upsample the chrominance very
> sensibly. However, their software goes back a decade or so, so I don't
> know if it has been superseded or abandoned by now.
>
> John
> 

In comp.compression John Costella <jpcostella@hotmail.com> wrote:
>
>>> http://www.assassinationscience.com/johncostella/unblock
>
> The best existing results I could find on the net were from Pegasus
> Imaging, which also deblocked without needing user input or fine
> tuning, but their decompressor seemed to leave in some strange blocks
> and other artifacts, and didn't seem to upsample the chrominance
> very sensibly. However, their software goes back a decade or so,
> so I don't know if it has been superseded or abandoned by now.

Did you try PaintShopPro?  From the samples you posted in demo.html
your software works better.  But I haven't tested PSP de-blocking
since version 7; it might have improved by version 9.

I'll do a PSP 9 comparison and post results if you want.

Hi Jack,

My comparisons were based on the Minerva program, which I had
downloaded from pegasusimaging.com. You are right: the About box lists
it as Copyright 1996-2005, Version 1.0.0.155 for PIC version 2.00
libraries, and the page http://www.pegasusimaging.com/jpegfeatures.htm
refers to it as your current "Pegasus imaging viewer". Apologies for
being cautious as to whether it represented your current technology.

I have now downloaded JPEG Wizard 2, but it seems to do a much worse
job at deblocking than Minerva. Indeed, it only seems to apply
deblocking to very highly compressed images, and only then in selected
areas. I cannot find any option to turn on deblocking for less heavily
compressed images. Am I missing something?

I do not see the chrominance smoothing option either, in either Minerva
or JPEG Wizard 2; could you guide me to it?

In any case, the results from Minerva don't seem to be optimal. I have
compared them for the same test images as on the UnBlock demo page and
you can see from a side-by-side comparison that they fall short at all
compression ratios:

http://www.assassinationscience.com/johncostella/unblock/pegasus-v-unblock.html

I agree that the Pegasus decompressor does not introduce blurring or
distortion in its deblocking, but neither does UnBlock, and neither, I
assume, do other deblocking filters available.

I will continue to play with JPEG Wizard 2, but so far I am hitting a
brick wall with it.

Thanks for your help,
John

Thanks Marco. You might be right about that.

All but one of the papers on that site were no longer there, but the
one that was had a couple of sample grayscale results. They did look
quite good at low resolution, although on magnification remnant
blocking artifacts remained. And as you point out, it is a very slow
and inelegant way to do things. (58 multiplications and 254 additions
per pixel is a very high price to pay.)

They claim that the complexity of their algorithm is low, but I
disagree.

Thanks again,
John