Non-uniform Deblurring for Shaken Images

Oliver Whyte Josef Sivic Andrew Zisserman Jean Ponce

Try the online demo: http://willow-fd.rocq.inria.fr/unshake/

Overview

In general, blur resulting from camera shake is mostly due to the 3D rotation of the camera, causing a blur that can be significantly non-uniform across the image. However, most previous deblurring methods model the observed image as a convolution of a sharp image with a uniform blur kernel. We propose a new model of the blurring process and apply this model in the context of two different algorithms for camera shake removal, showing that our approach makes it possible to model and remove camera shake more effectively than previous methods.

Blur Model

In the paper, we show that the blur arising from camera shake is caused mainly by the 3D rotation of the camera while the shutter is open. For a perspective camera, we know that rotation of the camera leads to a 2D projective transformation of the image, as shown in image (a) below, which displays the apparent motion of scene points under two simple rotations of the camera. Note that in both cases, the motion clearly varies across the image.

Traditionally, blur has been modelled as a convolution of a sharp image with a blur kernel, however this model is clearly insufficient to describe the effects seen below. Convolution models a blurry image as a weighted sum of translated versions of the sharp image, whereas we know that in fact, while the shutter is open, the camera in fact sees a sequence of projectively-transformed versions of the sharp image. Images (b) & (c) below show the difference between these two models.

• 

  (a) Apparent motion of scene points
  under camera rotations

• 

  (b) Blur by convolution

• 

  (c) Blur by rotation of camera

In the same way that a convolution kernel can be considered as a set of weights used to sum up translated versions of the sharp image, we define a blur kernel for our model to be the set of weights used to sum up projectively-transformed versions of the sharp image. The parameter space for camera rotations is 3D, giving us 3D blur kernels, as shown below. This kernel in fact describes a different blur for each pixel of the image, as shown below, where the point spread functions (PSFs) for several locations are plotted on top of the blurry image itself.

• 

  Camera axes

• 

  The weights, or "blur kernel" defined over orientations of the camers

• 

  A blurry image, with the equivalent PSFs overlaid at several locations

Applications

In order to demonstrate the effectiveness of our model, we substitute it into two existing algorithms for removing camera shake, in place of their convolutional blur models, and show superior results and the ability to handle highly non-uniform blurs.

Blind Deblurring

We have adapted the camera shake-removal algorithm of Fergus et al. [1], based on the variational deblurring method of Miskin & MacKay [2], to use our rotational blur model.

• 

  Blurry image

• 

  Deblurred using uniform blur model

• 

  Deblurred using our model

We have also adapted the fast blind deblurring algorithm of Cho & Lee [4] to use our rotational blur model.

• 

  Blurry image

• 

  Deblurred using uniform blur model

• 

  Deblurred using our model

Noisy/Blurry Image Pairs

Following the approach of Yuan et al. [3] we also apply our model in the situation where an additional sharp but noisy image of the same scene is available. This enables us to find a good estimate of the kernel with a standard convex optimization algorithm, and obtain a deblurred result free from "ringing" artifacts.

• 

  Blurry image

• 

  Noisy image

• 

  Deblurred using uniform blur model

• 

  Deblurred using our model

Publications

@inproceedings{whyte10,
  author = {Whyte, O. and Sivic, J. and Zisserman, A. and Ponce, J.},
  title = {Non-uniform Deblurring for Shaken Images},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year = {2010}
}

@article{whyte11a,
  author = {Whyte, O. and Sivic, J. and Zisserman, A. and Ponce, J.},
  title = {Non-uniform Deblurring for Shaken Images},
  journal = {International Journal of Computer Vision},
  volume = {98},
  number = {2},
  pages = {168--186},
  year = {2012}
}

Images

The images from the CVPR 2010 paper: Download (36MB)

Code

A package of Matlab code for removing non-uniform camera shake blur from a single blurry image. The code is based on that of Fergus et al. [1], and Miskin & MacKay [2].

Version 0.2.1 (30-Sep-2010): Download (2MB) | Readme
Version 0.1 (30-Aug-2010): Download (2MB) | Readme

Code

This package contains code to perform fast blind deblurring of images degraded by camera shake, using the MAP algorithm described in our IJCV 2012 paper, and the fast approximation of spatially-varying blur described in our CPCV 2011 paper.

Version 1.0 (21-Sep-2014): Download (18MB) | Readme

Online Demo

Try deblurring your own images with our online demo of the MAP-ℓ1 algorithm described in the IJCV article.
The demo is available at: http://willow-fd.rocq.inria.fr/unshake/.

Acknowledgements

This work was supported by ONR MURI N00014-07-1-0182, ERC grant VisRec no. 228180, the MSR-INRIA laboratory, ANR grant HFIBMR (ANR-07-BLAN-0331-01), the EIT-ICT labs (activity 10863), and the ERC grant VideoWorld.

References

[1] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman. Removing camera shake from a single photograph. SIGGRAPH, 2006.

[2] J. W. Miskin and D. J. C. MacKay. Ensemble learning for blind image separation and deconvolution. In Advances in Independent Component Analysis. Springer-Verlag, 2000.

[3] L. Yuan, J. Sun, L. Quan, and H.-Y. Shum. Image deblurring with blurred/noisy image pairs. SIGGRAPH, 2007.

[4] S. Cho and S. Lee. Fast Motion Deblurring. SIGGRAPH Asia, 2009.

Last updated: 4th June 2012