Overview

In this project, we tackle the problem of inpainting, or image completion, using images of the same scene retrieved from the Internet using viewpoint-invariant image search. The images below show an example result from our system.

  • Query Image

    Query Image

  • Target Regions

    Target Regions

  • Our Result

    Result

  • Retrieved images

    Some of the images retrieved from the Internet, used to produce the result

Method

The system takes as inputs a query image, and a target region within that image that we wish to replace. We begin by using a viewpoint-invariant image search engine (in our case the online Oxford Building Search Demo) to find other images of the same scene. We register each of these retrieved images to our query image with multiple homographies, apply a global photometric correction, and use each to propose a solution by combining it into the target region using Poisson blending. In the final step, we use a multi-label MRF formulation to decide which proposal to use at each pixel, to obtain a convincing, occlusion-free composite result.

Retrieving Oracles

Using a viewpoint-invariant image search engine, we retrieve many other images of the same scene and take the top 30 search results to be our oracles. Here we show 5 of these automatically retrieved oracles for the query image shown above:

Geometric Registration

The oracles are registered to the query image with homographies, allowing registration of multiple scene planes, or images taken from the same viewpoint:

Generating Proposals

Each registered oracle is used to propose a solution, by combining it into the target region using Poisson blending, to minimise the effect of differences in lighting:

Combining Multiple Proposals

In the final step, an MRF formulation allows the multiple proposals to be combined into a single result, selecting reliable regions of different proposals and concealing the boundaries between regions:

  • Query Image

    Query Image

  • Output Labels

    Output Labels

  • Our Result

    Result

Paper

O. Whyte, J. Sivic and A. Zisserman
Get Out of my Picture! Internet-based Inpainting
Proceedings of the 20th British Machine Vision Conference, London, September 2009, oral
PDF | Abstract | BibTeX

Abstract

We present a method to replace a user specified target region of a photograph by using other photographs of the same scene downloaded from the Internet via viewpoint-invariant image search. Each of the retrieved images is first geometrically then photometrically registered with the query photograph. Geometric registration is achieved using multiple homographies and photometric registration is performed using a global affine transformation on image intensities. Each retrieved image proposes a possible solution for the target region. In the final step we combine these proposals into a single visually consistent result, using a Markov random field optimisation to choose seams between proposals, followed by gradient domain fusion. We demonstrate removal of objects and people in challenging photographs of Oxford landmarks containing complex image structures.

BibTeX

@inproceedings{WSZ09,
  author = {Oliver Whyte and Josef Sivic and Andrew Zisserman},
  title = {Get Out of my Picture! Internet-based Inpainting},
  booktitle = {Proceedings of the 20th British Machine Vision Conference, London},
  year = {2009}
}

Additional Results

Mouseover the images to see the results, click to enlarge...