Action Anticipation from Multimodal Data

Tiziana Rotondo, Giovanni Maria Farinella, Valeria Tomaselli, Sebastiano Battiato

2019

Abstract

The idea of multi-sensor data fusion is to combine the data coming from different sensors to provide more accurate and complementary information to solve a specific task. Our goal is to build a shared representation related to data coming from different domains, such as images, audio signal, heart rate, acceleration, etc., in order to anticipate daily activities of a user wearing multimodal sensors. To this aim, we consider the Stanford-ECM Dataset which contains syncronized data acquired with different sensors: video, acceleration and heart rate signals. The dataset is adapted to our action prediction task by identifying the transitions from the generic “Unknown” class to a specific “Activity”. We discuss and compare a Siamese Network with the Multi Layer Perceptron and the 1D CNN where the input is an unknown observation and the output is the next activity to be observed. The feature representations obtained with the considered deep architecture are classified with SVM or KNN classifiers. Experimental results pointed out that prediction from multimodal data seems a feasible task, suggesting that multimodality improves both classification and prediction. Nevertheless, the task of reliably predicting next actions is still open and requires more investigations as well as the availability of multimodal dataset, specifically built for prediction purposes.

Paper Citation

in Harvard Style

Rotondo T., Farinella G., Tomaselli V. and Battiato S. (2019). Action Anticipation from Multimodal Data. In Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019) - Volume 4: VISAPP; ISBN 978-989-758-354-4, SciTePress, pages 154-161. DOI: 10.5220/0007379001540161

in Bibtex Style

@conference{visapp19,
author={Tiziana Rotondo and Giovanni Maria Farinella and Valeria Tomaselli and Sebastiano Battiato},
title={Action Anticipation from Multimodal Data},
booktitle={Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019) - Volume 4: VISAPP},
year={2019},
pages={154-161},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0007379001540161},
isbn={978-989-758-354-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019) - Volume 4: VISAPP
TI - Action Anticipation from Multimodal Data
SN - 978-989-758-354-4
AU - Rotondo T.
AU - Farinella G.
AU - Tomaselli V.
AU - Battiato S.
PY - 2019
SP - 154
EP - 161
DO - 10.5220/0007379001540161
PB - SciTePress