Unsupervised Gaze Prediction in Egocentric Videos by Energy-based Surprise Modeling

Sathyanarayanan N. Aakur, Arunkumar Bagavathi

2021

Abstract

Egocentric perception has grown rapidly with the advent of immersive computing devices. Human gaze prediction is an important problem in analyzing egocentric videos and has primarily been tackled through either saliency-based modeling or highly supervised learning. We quantitatively analyze the generalization capabilities of supervised, deep learning models on the egocentric gaze prediction task on unseen, out-of-domain data. We find that their performance is highly dependent on the training data and is restricted to the domains specified in the training annotations. In this work, we tackle the problem of jointly predicting human gaze points and temporal segmentation of egocentric videos without using any training data. We introduce an unsupervised computational model that draws inspiration from cognitive psychology models of event perception. We use Grenander’s pattern theory formalism to represent spatial-temporal features and model surprise as a mechanism to predict gaze fixation points. Extensive evaluation on two publicly available datasets - GTEA and GTEA+ datasets-shows that the proposed model can significantly outperform all unsupervised baselines and some supervised gaze prediction baselines. Finally, we show that the model can also temporally segment egocentric videos with a performance comparable to more complex, fully supervised deep learning baselines.

Paper Citation

in Harvard Style

Aakur S. and Bagavathi A. (2021). Unsupervised Gaze Prediction in Egocentric Videos by Energy-based Surprise Modeling. In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP; ISBN 978-989-758-488-6, SciTePress, pages 935-942. DOI: 10.5220/0010288009350942

in Bibtex Style

@conference{visapp21,
author={Sathyanarayanan N. Aakur and Arunkumar Bagavathi},
title={Unsupervised Gaze Prediction in Egocentric Videos by Energy-based Surprise Modeling},
booktitle={Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP},
year={2021},
pages={935-942},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010288009350942},
isbn={978-989-758-488-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP
TI - Unsupervised Gaze Prediction in Egocentric Videos by Energy-based Surprise Modeling
SN - 978-989-758-488-6
AU - Aakur S.
AU - Bagavathi A.
PY - 2021
SP - 935
EP - 942
DO - 10.5220/0010288009350942
PB - SciTePress