Hybrid-S2S: Video Object Segmentation with Recurrent Networks and Correspondence Matching

Fatemeh Azimi, Fatemeh Azimi, Stanislav Frolov, Stanislav Frolov, Federico Raue, Jörn Hees, Andreas Dengel, Andreas Dengel

2021

Abstract

One-shot Video Object Segmentation (VOS) is the task of pixel-wise tracking an object of interest within a video sequence, where the segmentation mask of the first frame is given at inference time. In recent years, Recurrent Neural Networks (RNNs) have been widely used for VOS tasks, but they often suffer from limitations such as drift and error propagation. In this work, we study an RNN-based architecture and address some of these issues by proposing a hybrid sequence-to-sequence architecture named HS2S, utilizing a dual mask propagation strategy that allows incorporating the information obtained from correspondence matching. Our experiments show that augmenting the RNN with correspondence matching is a highly effective solution to reduce the drift problem. The additional information helps the model to predict more accurate masks and makes it robust against error propagation. We evaluate our HS2S model on the DAVIS2017 dataset as well as Youtube-VOS. On the latter, we achieve an improvement of 11.2pp in the overall segmentation accuracy over RNN-based state-of-the-art methods in VOS. We analyze our model’s behavior in challenging cases such as occlusion and long sequences and show that our hybrid architecture significantly enhances the segmentation quality in these difficult scenarios.

Paper Citation

in Harvard Style

Azimi F., Frolov S., Raue F., Hees J. and Dengel A. (2021). Hybrid-S2S: Video Object Segmentation with Recurrent Networks and Correspondence Matching. In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 4: VISAPP; ISBN 978-989-758-488-6, SciTePress, pages 182-192. DOI: 10.5220/0010339401820192

in Bibtex Style

@conference{visapp21,
author={Fatemeh Azimi and Stanislav Frolov and Federico Raue and Jörn Hees and Andreas Dengel},
title={Hybrid-S2S: Video Object Segmentation with Recurrent Networks and Correspondence Matching},
booktitle={Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 4: VISAPP},
year={2021},
pages={182-192},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010339401820192},
isbn={978-989-758-488-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 4: VISAPP
TI - Hybrid-S2S: Video Object Segmentation with Recurrent Networks and Correspondence Matching
SN - 978-989-758-488-6
AU - Azimi F.
AU - Frolov S.
AU - Raue F.
AU - Hees J.
AU - Dengel A.
PY - 2021
SP - 182
EP - 192
DO - 10.5220/0010339401820192
PB - SciTePress