Chuan Wang

Content-Aware Unsupervised Deep Homography Estimation

Jirong Zhang1,2,*  Chuan Wang2,*  Shuaicheng Liu1,2,#  Lanpeng Jia2  Nianjin Ye2  Jue Wang2  Ji Zhou1  Jian Sun2 

1University of Electronic Science and Technology of China                      2Megvii Technology

*Joint First Authors                      #Corresponding Author

The 16th European Conference on Computer Vision (ECCV 2020), Oral

arXiv https://arxiv.org/pdf/1909.05983.pdf

deephomo Figure. Our deep homography estimation on challenging cases, compared with one traditional feature-based, i.e. SIFT + RANSAC and one unsupervised DNN-based method. (a) An example with dominate moving foreground. (b) A low texture example. (c) A low light example. We mix the blue and green channels of the warped image and the red channel of the target image to obtain the visualization results as above, where the misaligned pixels appear as red or green ghosts. The same visualization method is applied for the rest of this paper.

Abstract

Homography estimation is a basic image alignment method in many applications. It is usually conducted by extracting and matching sparse feature points, which are error-prone in low-light and low-texture images. On the other hand, previous deep homography approaches use either synthetic images for supervised learning or aerial images for unsupervised learning, both ignoring the importance of handling depth disparities and moving objects in real world applications. To overcome these problems, in this work we propose an unsupervised deep homography method with a new architecture design. In the spirit of the RANSAC procedure in traditional methods, we specifically learn an outlier mask to only select reliable regions for homography estimation. We calculate loss with respect to our learned deep features instead of directly comparing image content as did previously. To achieve the unsupervised training, we also formulate a novel triplet loss customized for our network. We verify our method by conducting comprehensive comparisons on a new dataset that covers a wide range of scenes with varying degrees of difficulties for the task. Experimental results reveal that our method outperforms the state-of-the-art including deep solutions and feature-based solutions.

Downloads

    paper  
    source code  

Video Demo

    Bilibili  

Bibtex

@inproceedings{zhang2020content,
  title={Content-aware unsupervised deep homography estimation},
  author={Zhang, Jirong and Wang, Chuan and Liu, Shuaicheng and Jia, Lanpeng and Ye, Nianjin and Wang, Jue and Zhou, Ji and Sun, Jian},
  booktitle={European Conference on Computer Vision},
  pages={653--669},
  year={2020},
  organization={Springer}
}