Try the online demo: http://willowfd.rocq.inria.fr/unshake/
Overview
In general, blur resulting from camera shake is mostly due to the 3D rotation of the camera, causing a blur that can be significantly nonuniform across the image. However, most previous deblurring methods model the observed image as a convolution of a sharp image with a uniform blur kernel. We propose a new model of the blurring process and apply this model in the context of two different algorithms for camera shake removal, showing that our approach makes it possible to model and remove camera shake more effectively than previous methods.
Blur Model
In the paper, we show that the blur arising from camera shake is caused mainly by the 3D rotation of the camera while the shutter is open. For a perspective camera, we know that rotation of the camera leads to a 2D projective transformation of the image, as shown in image (a) below, which displays the apparent motion of scene points under two simple rotations of the camera. Note that in both cases, the motion clearly varies across the image.
Traditionally, blur has been modelled as a convolution of a sharp image with a blur kernel, however this model is clearly insufficient to describe the effects seen below. Convolution models a blurry image as a weighted sum of translated versions of the sharp image, whereas we know that in fact, while the shutter is open, the camera in fact sees a sequence of projectivelytransformed versions of the sharp image. Images (b) & (c) below show the difference between these two models.

(a) Apparent motion of scene points
under camera rotations 
(b) Blur by convolution

(c) Blur by rotation of camera
In the same way that a convolution kernel can be considered as a set of weights used to sum up translated versions of the sharp image, we define a blur kernel for our model to be the set of weights used to sum up projectivelytransformed versions of the sharp image. The parameter space for camera rotations is 3D, giving us 3D blur kernels, as shown below. This kernel in fact describes a different blur for each pixel of the image, as shown below, where the point spread functions (PSFs) for several locations are plotted on top of the blurry image itself.

Camera axes

The weights, or "blur kernel" defined over orientations of the camers

A blurry image, with the equivalent PSFs overlaid at several locations
Applications
In order to demonstrate the effectiveness of our model, we substitute it into two existing algorithms for removing camera shake, in place of their convolutional blur models, and show superior results and the ability to handle highly nonuniform blurs.
Blind Deblurring
We have adapted the camera shakeremoval algorithm of Fergus et al. [1], based on the variational deblurring method of Miskin & MacKay [2], to use our rotational blur model.
We have also adapted the fast blind deblurring algorithm of Cho & Lee [4] to use our rotational blur model.
Noisy/Blurry Image Pairs
Following the approach of Yuan et al. [3] we also apply our model in the situation where an additional sharp but noisy image of the same scene is available. This enables us to find a good estimate of the kernel with a standard convex optimization algorithm, and obtain a deblurred result free from "ringing" artifacts.
Publications
Nonuniform Deblurring for Shaken Images
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2010)
PDF  Abstract  BibTeX  PPT Slides (20MB)  Poster (8MB)  Derivation of parameter update equations for blind deblurring
Abstract
Blur from camera shake is mostly due to the 3D rotation of the camera, resulting in a blur kernel that can be significantly nonuniform across the image. However, most current deblurring methods model the observed image as a convolution of a sharp image with a uniform blur kernel. We propose a new parametrized geometric model of the blurring process in terms of the rotational velocity of the camera during exposure. We apply this model to two different algorithms for camera shake removal: the first one uses a single blurry image (blind deblurring), while the second one uses both a blurry image and a sharp but noisy image of the same scene. We show that our approach makes it possible to model and remove a wider class of blurs than previous approaches, including uniform blur as a special case, and demonstrate its effectiveness with experiments on real images.BibTeX
@inproceedings{whyte10, author = {Whyte, O. and Sivic, J. and Zisserman, A. and Ponce, J.}, title = {Nonuniform Deblurring for Shaken Images}, booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, year = {2010} }
Nonuniform Deblurring for Shaken Images
International Journal of Computer Vision, 98(2):168–186, 2012
PDF  Abstract  BibTeX  Journal page
Abstract
Photographs taken in lowlight conditions are often blurry as a result of camera shake, i.e. a motion of the camera while its shutter is open. Most existing deblurring methods model the observed blurry image as the convolution of a sharp image with a uniform blur kernel. However, we show that blur from camera shake is in general mostly due to the 3D rotation of the camera, resulting in a blur that can be significantly nonuniform across the image. We propose a new parametrized geometric model of the blurring process in terms of the rotational motion of the camera during exposure. This model is able to capture nonuniform blur in an image due to camera shake using a single global descriptor, and can be substituted into existing deblurring algorithms with only small modifications. To demonstrate its effectiveness, we apply this model to two deblurring problems; first, the case where a single blurry image is available, for which we examine both an approximate marginalization approach and a maximum a posteriori approach, and second, the case where a sharp but noisy image of the scene is available in addition to the blurry image. We show that our approach makes it possible to model and remove a wider class of blurs than previous approaches, including uniform blur as a special case, and demonstrate its effectiveness with experiments on synthetic and real images.BibTeX
@article{whyte11a, author = {Whyte, O. and Sivic, J. and Zisserman, A. and Ponce, J.}, title = {Nonuniform Deblurring for Shaken Images}, journal = {International Journal of Computer Vision}, volume = {98}, number = {2}, pages = {168186}, year = {2012} }
Images
The images from the CVPR 2010 paper: Download (36MB)
Code
A package of Matlab code for removing nonuniform camera shake blur from a single blurry image. The code is based on that of Fergus et al. [1], and Miskin & MacKay [2].
Version 0.2.1 (30Sep2010): Download (2MB)  Readme
Version 0.1 (30Aug2010): Download (2MB)  Readme
Code
This package contains code to perform fast blind deblurring of images degraded by camera shake, using the MAP algorithm described in our IJCV 2012 paper, and the fast approximation of spatiallyvarying blur described in our CPCV 2011 paper.
Version 1.0 (21Sep2014): Download (18MB)  Readme
Online Demo
Try deblurring your own images with our online demo of the MAPℓ1 algorithm described in the IJCV article.
The demo is available at: http://willowfd.rocq.inria.fr/unshake/.
Acknowledgements
This work was supported by ONR MURI N000140710182, ERC grant VisRec no. 228180, the MSRINRIA laboratory, ANR grant HFIBMR (ANR07BLAN033101), the EITICT labs (activity 10863), and the ERC grant VideoWorld.
References
[1] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman. Removing camera shake from a single photograph. SIGGRAPH, 2006.
[2] J. W. Miskin and D. J. C. MacKay. Ensemble learning for blind image separation and deconvolution. In Advances in Independent Component Analysis. SpringerVerlag, 2000.
[3] L. Yuan, J. Sun, L. Quan, and H.Y. Shum. Image deblurring with blurred/noisy image pairs. SIGGRAPH, 2007.
[4] S. Cho and S. Lee. Fast Motion Deblurring. SIGGRAPH Asia, 2009.