### Sparse Neighbourhood Consensus Networks

Given an input image pair (left), we show the raw output correspondences produced by Sparse-NCNet (center) which contain groups of spatially coherent matches. These groups tend to form around highly-confident matches, which are shown in yellow shades (right).

### Abstract

In this work we target the problem of estimating accurately localised correspondences between a pair of images. We adopt the recent Neighbourhood Consensus Networks that have demonstrated promising performance for difficult correspondence problems and propose modifications to overcome their main limitations: large memory consumption, large inference time and poorly localised correspondences. Our proposed modifications can reduce the memory footprint and execution time more than $10\times$, with equivalent results. This is achieved by sparsifying the correlation tensor containing tentative matches, and its subsequent processing with a 4D CNN using submanifold sparse convolutions. Localisation accuracy is significantly improved by processing the input images in higher resolution, which is possible due to the reduced memory footprint, and by a novel two-stage correspondence relocalisation module. The proposed Sparse-NCNet method obtains state-of-the-art results on the HPatches Sequences and InLoc visual localisation benchmarks, and competitive results in the Aachen Day-Night benchmark.

### Paper

I. Rocco, R. Arandjelović and J. Sivic
Efficient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions
European Conference on Computer Vision, 2020
[Paper on arXiv]

#### BibTeX

@inproceedings{Rocco20,
author       = "Rocco, I. and Arandjelovi\'c, R. and Sivic, J.",
title        = "Efficient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions",
booktitle    = "European Conference on Computer Vision",
year         = 2020,
}

### Code

Please see the Github repo here: https://github.com/ignacio-rocco/sparse-ncnet.

### Acknowledgements

This work was partially supported by ERC grant LEAP No. 336845, the European Regional Development Fund under project IMPACT (reg. no. CZ.02.1.01/0.0/0.0/15 003/0000468), Louis Vuitton ENS Chair on Artificial Intelligence, and the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute).