Try the online demo: http://willow-fd.rocq.inria.fr/unshake/

Overview

One common feature of “shaken” images is the presence of saturated pixels. These are caused when the radiance of the scene exceeds the range of the camera’s sensor, leaving bright highlights clipped at the maximum output value (e.g. 255 for an 8-bit image). To anyone who has attempted to take hand-held photographs at night, this effect should be familiar as the conspicuous bright streaks left by electric lights, such as in the images below. These bright pixels, with their clipped values, violate the assumption made by many algorithms that the image formation process is linear, and as a result can cause obtrusive artifacts in the deblurred images. This can be seen in the example images below. In this paper, we propose a non-blind deblurring algorithm that takes account of saturated pixels, and is able to greatly reduce the artifacts they cause in the deblurred results.


We also demonstrate an efficient approximation for spatially-varying blur, extending the work of Hirsch et al. [4] to handle our previously-proposed global model for spatially-varying camera shake blur [5]. This approximation, based on the local uniformity of the blur, allows the forward model for spatially-varying blur to be computed significantly faster than the exact model, while retaining the benefits of the global parameterization. Note that independently of our work, Hirsch et al. [6] have proposed a similar method to combine a global model of spatially-varying blur with their locally-uniform approximation.

Examples

Click to enlarge images. Navigate through with cursor keys.

  • Blurry image with saturation

  • Deblurred with Richardson-Lucy [1,2]

  • Deblurred with algorithm of Krishnan & Fergus [3]

  • Deblurred with proposed method

  • Blurry image with saturation
  • Deblurred with Richardson-Lucy [1,2]
  • Deblurred with algorithm of Krishnan & Fergus [3]
  • Deblurred with proposed method
  • Blurry image with saturation
  • Deblurred with Richardson-Lucy [1,2]
  • Deblurred with algorithm of Krishnan & Fergus [3]
  • Deblurred with proposed method
  • Blurry image with saturation
  • Deblurred with Richardson-Lucy [1,2]
  • Deblurred with algorithm of Krishnan & Fergus [3]
  • Deblurred with proposed method
  • Blurry image with saturation
  • Deblurred with Richardson-Lucy [1,2]
  • Deblurred with algorithm of Krishnan & Fergus [3]
  • Deblurred with proposed method

Efficient Approximation of the Global Blur Model

The images below show a synthetic blur kernel for our previously proposed model [5]. The PSF is visualized at various points across the image, using the exact forward model and the approximation at increasingly fine levels of discretization. The approximation quickly approaches the exact model, and at the finest discretization shown it is almost identical to the exact model.

  • Global blur kernel

    Global blur kernel

  • Exact forward model

    PSF, exact forward model

  • Approximate forward model, 3 × 4 patches

    PSF, approximate forward model,
    3 × 4 patches

  • Approximate forward model, 6 × 8 patches

    PSF, approximate forward model,
    6 × 8 patches

  • Approximate forward model, 12 × 16 patches

    PSF, approximate forward model,
    12 × 16 patches

Paper

O. Whyte, J. Sivic and A. Zisserman
Deblurring Shaken and Partially Saturated Images
IEEE Color and Photometry in Computer Vision Workshop (2011), in conjunction with ICCV 2011
PDF | Abstract | BibTeX | PPT Slides (19MB)

Abstract

We address the problem of deblurring images degraded by camera shake blur and saturated or over-exposed pixels. Saturated pixels are a problem for existing non-blind deblurring algorithms because they violate the assumption that the image formation process is linear, and often cause significant artifacts in deblurred outputs. We propose a forward model that includes sensor saturation, and use it to derive a deblurring algorithm properly treating saturated pixels. By using this forward model and reasoning about the causes of artifacts in the deblurred results, we obtain significantly better results than existing deblurring algorithms. Further we propose an efficient approximation of the forward model leading to a significant speed-up.

BibTeX

@inproceedings{whyte11,
  author = {Oliver Whyte and Josef Sivic and Andrew Zisserman},
  title = {Deblurring Shaken and Partially Saturated Images}
  booktitle = {Proceedings of the IEEE Workshop on Color and Photometry in Computer Vision, with ICCV 2011},
  year = {2011},
}
O. Whyte, J. Sivic and A. Zisserman
Deblurring Shaken and Partially Saturated Images
International Journal of Computer Vision, 2014
PDF | Abstract | BibTeX | Journal page

Abstract

We address the problem of deblurring images degraded by camera shake blur and saturated (over-exposed) pixels. Saturated pixels violate the common assumption that the image-formation process is linear, and often cause ringing in deblurred outputs. We provide an analysis of ringing in general, and show that in order to prevent ringing, it is insufficient to simply discard saturated pixels. We show that even when saturated pixels are removed, ringing is caused by attempting to estimate the values of latent pixels that are brighter than the sensor’s maximum output. Estimating these latent pixels is likely to cause large errors, and these errors propagate across the rest of the image in the form of ringing. We propose a new deblurring algorithm that locates these error-prone bright pixels in the latent sharp image, and by decoupling them from the remainder of the latent image, greatly reduces ringing. In addition, we propose an approximate forward model for saturated images, which allows us to estimate these error-prone pixels separately without causing artefacts. Results are shown for non-blind deblurring of real photographs containing saturated regions, demonstrating improved deblurred image quality compared to previous work.

BibTeX

@article{whyte14,
  author = {Whyte, O. and Sivic, J. and Zisserman, A.},
  title = {Deblurring Shaken and Partially Saturated Images},
  journal = {International Journal of Computer Vision},
  doi={10.1007/s11263-014-0727-3},
  url={http://dx.doi.org/10.1007/s11263-014-0727-3},
  year = {2014}
}}

Images

The images from the CPCV 2011 paper: Download (25MB)

The results of our algorithm on the images of Cho et al. [7]: Download (16MB)

Code

A package of Matlab code for non-blind deblurring of blurry images with clipped / saturated pixels.
Version 0.1 (20-Nov-2011): Download (7MB) | Readme

Code

This package contains code to perform fast blind deblurring of images degraded by camera shake, using the MAP algorithm described in our IJCV 2012 paper, and the fast approximation of spatially-varying blur described in our CPCV 2011 paper.

Version 1.0 (21-Sep-2014): Download (18MB) | Readme

Comparison to the Method of Cho et al.

The images below show the results of our non-blind deblurring algorithm on images provided by Cho et al. [7]. The (spatially-invariant) blur kernels for the images, along with the results of their algorithm, are provided by the authors online here.

Download the uncompressed images above, or at this link.

Click to enlarge images. Navigate through with cursor keys.

  • Blurry Image

  • Result from Cho et al. [7]

  • Our result

  • Blurry Image
  • Result of Cho et al.
  • Our result
  • Blurry Image
  • Result of Cho et al.
  • Our result
  • Blurry Image
  • Result of Cho et al.
  • Our result
  • Blurry Image
  • Result of Cho et al.
  • Our result
  • Blurry Image
  • Result of Cho et al.
  • Our result
  • Blurry Image
  • Result of Cho et al.
  • Our result

References

[1] W. H. Richardson. Bayesian-Based Iterative Method of Image Restoration. Journal of the Optical Society of America, 62(1), 1972.

[2] L. B. Lucy. An iterative technique for the rectification of observed distributions. Astronomical Journal, 79(6), 1974.

[3] D. Krishnan and R. Fergus. Fast Image Deconvolution using Hyper-Laplacian Priors. In Proc. NIPS, 2009.

[4] M. Hirsch, S. Sra, B. Scholkopf, and S. Harmeling. Efficient Filter Flow for Space-Variant Multiframe Blind Deconvolution. In Proc. CVPR, 2010.

[5] O. Whyte, J. Sivic, A. Zisserman, and J. Ponce. Non-uniform Deblurring for Shaken Images. In Proc. CVPR, 2010.

[6] M. Hirsch, C. J. Schuler, S. Harmeling, and B. Schölkopf. Fast Removal of Non-uniform Camera Shake. In Proc. ICCV, 2011.

[7] S. Cho, J. Wang, and S. Lee. Handling Outliers in Non-blind Image Deconvolution. In Proc. ICCV, 2011.