STIFS: Spatio-Temporal Input Frame Selection for Learning-based Video Super-Resolution Models

Arbind Agrahari Baniya, Tsz-Kwan Lee, Peter Eklund, Sunil Aryal

2022

Abstract

Deep learning Video Super-Resolution (VSR) methods rely on learning spatio-temporal correlations between a target frame and its neighbouring frames in a given temporal radius to generate a high-resolution output. Among recent VSR models, a sliding window mechanism is popularly adopted by picking a fixed number of consecutive frames as neighbouring frames for a given target frame. This results in a single frame being used multiple times in the input space during the super-resolution process. Moreover, the approach of adopting the fixed consecutive frames directly does not allow deep learning models to learn the full extent of spatio-temporal inter-dependencies between a target frame and its neighbours along a video sequence. To mitigate these issues, this paper proposes a Spatio-Temporal Input Frame Selection (STIFS) algorithm based on image analysis to adaptively select the neighbouring frame(s) based on the spatio-temporal context dynamics with respect to the target frame. STIFS is first-ever dynamic selection mechanism proposed for VSR methods. It aims to enable VSR models to better learn spatio-temporal correlations in a given temporal radius and consequently maximise the quality of the high-definition output. The proposed STIFS algorithm achieved remarkable PSNR improvements in the high-resolution output for VSR models on benchmark datasets.

Paper Citation

in Harvard Style

Agrahari Baniya A., Lee T., Eklund P. and Aryal S. (2022). STIFS: Spatio-Temporal Input Frame Selection for Learning-based Video Super-Resolution Models. In Proceedings of the 19th International Conference on Signal Processing and Multimedia Applications - Volume 1: SIGMAP, ISBN 978-989-758-591-3, pages 48-58. DOI: 10.5220/0011339900003289

in Bibtex Style

@conference{sigmap22,
author={Arbind Agrahari Baniya and Tsz-Kwan Lee and Peter Eklund and Sunil Aryal},
title={STIFS: Spatio-Temporal Input Frame Selection for Learning-based Video Super-Resolution Models},
booktitle={Proceedings of the 19th International Conference on Signal Processing and Multimedia Applications - Volume 1: SIGMAP,},
year={2022},
pages={48-58},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011339900003289},
isbn={978-989-758-591-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 19th International Conference on Signal Processing and Multimedia Applications - Volume 1: SIGMAP,
TI - STIFS: Spatio-Temporal Input Frame Selection for Learning-based Video Super-Resolution Models
SN - 978-989-758-591-3
AU - Agrahari Baniya A.
AU - Lee T.
AU - Eklund P.
AU - Aryal S.
PY - 2022
SP - 48
EP - 58
DO - 10.5220/0011339900003289