Convolutional neural network architecture for geometric matching
Our trained geometry estimation network automatically aligns two images with substantial appearance differences. It is able to estimate large deformable transformations robustly in the presence of clutter.
The proposed CNN architecture is based on three main components that mimic the standard steps of feature extraction, matching and simultaneous inlier detection and model parameter estimation. A first stage achieves a rough alingment using an affine transformation, and a second stage refines this alignment using a thin-plate spline transformation.
Abstract
We address the problem of determining correspondences between two images in agreement with a geometric model such as an affine or thin-plate spline transformation, and estimating its parameters. The contributions of this work are three-fold. First, we propose a convolutional neural network architecture for geometric matching. The architecture is based on three main components that mimic the standard steps of feature extraction, matching and simultaneous inlier detection and model parameter estimation, while being trainable end-to-end. Second, we demonstrate that the network parameters can be trained from synthetically generated imagery without the need for manual annotation and that our matching layer significantly increases generalization capabilities to never seen before images. Finally, we show that the same model can perform both instance-level and category-level matching giving state-of-the-art results on the challenging Proposal Flow dataset.
Papers
|
I. Rocco, R. Arandjelović and J. Sivic Convolutional neural network architecture for geometric matching In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017 [Paper on arXiv][BibTeX] |
|
I. Rocco, R. Arandjelović and J. Sivic Convolutional neural network architecture for geometric matching In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018 [Paper on HAL] [BibTeX] |
Spotlight & poster @ CVPR'17
|
Spotlight presentation (4min) |
|
Poster |
Code
-
[MatConvNet code][PyTorch code]
: PyTorch models using ResNet-101 have been added (we thank Hao Zhao for this suggestion).
Acknowledgements
This work has been partly supported by ERC grant LEAP (no. 336845), ANR project Semapolis (ANR-13-CORD-0003), the Inria CityLab IPL, CIFAR Learning in Machines & Brains program and ESIF, OP Research, development and education Project IMPACT No. CZ.02.1.01/0.0/0.0/15 003/0000468.
Copyright Notice
The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright.