Automatic Alignment of Paintings and Photographs Depicting a 3D Scene

Bryan C. Russell, Josef Sivic, Jean Ponce, Hélène Dessales

3D model automatically obtained from photographs Input painting Automatic alignment of
painting to 3D model

Overview

This paper addresses the problem of automatically aligning historical architectural paintings with 3D models obtained using multi-view stereo technology from modern photographs. This is a challenging task because of the variations in appearance, geometry, color and texture due to environmental changes over time, the nonphotorealistic nature of architectural paintings, and differences in the viewpoints used by the painters and photographers. Our alignment procedure consists of two novel aspects: (i) we combine the gist descriptor with the view-synthesis/retrieval of Irschara et al. to obtain a coarse alignment of the painting to the 3D model, and (ii) we have developed an ICP-like viewpoint refinement procedure, where 3D surface orientation discontinuities (folds and creases) and view-dependent occlusion boundaries are rendered from the automatically obtained and noisy 3D model in a view-dependent manner and matched to gPB contours extracted from the paintings. We demonstrate the alignment of XIXth Century architectural watercolors of the Casa di Championnet in Pompeii with a 3D model constructed from modern photographs using the PMVS public-domain multi-view stereo software.


Paper

B. C. Russell, J. Sivic, J. Ponce, and H. Dessales, Automatic Alignment of Paintings and Photographs Depicting a 3D Scene, 3rd International IEEE Workshop on 3D Representation for Recognition (3dRR-11), associated with ICCV 2011. (PDF | BibTeX)

Videos

Overview presentation (youtube, 6m37s)
Fly-through results (.mov, 39.1MB)

Talk

PPT and PDF (.zip, 48.4MB)



Downloads


Code

Zip file (or pull directly from the Github repository)

Related work

Please see also a more recent work on larger scale painting-to-3D model alignment via discriminative visual elements.

M. Aubry, B. C. Russell, J. Sivic
Painting-to-3D Model Alignment Via Discriminative Visual Elements
ACM Transactions on Graphics 2014 (will be presented at SIGGRAPH 2014)
PDF | BibTeX | Project webpage



Other related work (courtesy of Yves Ubelmann)



Acknowledgments

We thank Soprintendenza archeologica di Napoli e Pompei, for access to the site and the possibility of study; Margareta Staub Gierow (Archäologisches Institut, Albert- Ludwigs-Universität, Freiburg), for her collaboration in collecting archives in the National Museum of Stockolm. Supported in part by the MSR-INRIA joint laboratory, the ANR project DETECT (ANR-09-JCJC-0027-01) and the EIT ICT labs (activity 10863).