Neighbourhood Consensus Networks

A fully convolutional neural network is used to extract dense image descriptors $f^A$ and $f^B$ for images $I_A$ and $I_B$, respectively. All pairs of individual feature matches $f^A_{ij}$ and $f^B_{kl}$ are represented in the 4-D space of matches $(i,j,k,l)$ (here shown as a 3-D perspective for illustration), and their matching scores stored in the 4-D correlation tensor $c$. These matches are further processed by the proposed soft-nearest neighbour filtering and neighbourhood consensus network to produce the final set of output correspondences.

Abstract

We address the problem of finding reliable dense correspondences between a pair of images. This is a challenging task due to strong appearance differences between the corresponding scene elements and ambiguities generated by repetitive patterns. The contributions of this work are threefold. First, inspired by the classic idea of disambiguating feature matches using semi-local constraints, we develop an end-to-end trainable convolutional neural network architecture that identifies sets of spatially consistent matches by analyzing neighbourhood consensus patterns in the 4D space of all possible correspondences between a pair of images without the need for a global geometric model. Second, we demonstrate that the model can be trained effectively from weak supervision in the form of matching and non-matching image pairs without the need for costly manual annotation of point to point correspondences. Third, we show the proposed neighbourhood consensus network can be applied to a range of matching tasks including both category- and instance-level matching, obtaining the state-of-the-art results on the PF Pascal dataset and the InLoc indoor visual localization benchmark.

Paper

I. Rocco, M. Cimpoi, R. Arandjelović, A. Torii, T. Pajdla and J. Sivic
Neighbourhood Consensus Networks
In Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018
[Paper on arXiv]

BibTeX

@InProceedings{Rocco18b,
        author       = "Rocco, I. and Cimpoi, M. and Arandjelovi\'c, R. and Torii, A. and Pajdla, T. and Sivic, J."
        title        = "Neighbourhood Consensus Networks",
        booktitle    = "Proceedings of the 32nd Conference on Neural Information Processing Systems",
        year         = "2018",
        }

Code and IVD dataset

Please see the Github repo here: https://github.com/ignacio-rocco/ncnet

Acknowledgements

This work was partially supported by JSPS KAKENHI Grant Numbers 15H05313, 16KK0002, EU-H2020 project LADIO No. 731970, ERC grant LEAP No. 336845, CIFAR Learning in Machines & Brains program and the European Regional Development Fund under the project IMPACT (reg. no. CZ.02.1.01/0.0/0.0/15 003/0000468).

Copyright Notice

The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright.