Authors:
Julius Schöning
1
;
Anna L. Gert
1
;
Alper Açık
2
;
Tim C. Kietzmann
3
;
Gunther Heidemann
1
and
Peter König
1
Affiliations:
1
Osnabrück University, Germany
;
2
Özyeğin University, Turkey
;
3
CB2 7EF and Medical Research Council, United Kingdom
Keyword(s):
Multimodal Data Analysis, Visualization, Sonification, Gaze Data, EEG Data.
Related
Ontology
Subjects/Areas/Topics:
Computer Vision, Visualization and Computer Graphics
;
Image and Video Coding and Compression
;
Image Formation and Preprocessing
;
Multimodal and Multi-Sensor Models of Image Formation
Abstract:
The analysis of multimodal data comprised of images, videos and additional recordings, such as gaze trajectories,
EEG, emotional states, and heart rate is presently only feasible with custom applications. Even exploring
such data requires compilation of specific applications that suit a specific dataset only. This need for specific
applications arises since all corresponding data are stored in separate files in custom-made distinct data formats.
Thus accessing such datasets is cumbersome and time-consuming for experts and virtually impossible
for non-experts. To make multimodal research data easily shareable and accessible to a broad audience, like
researchers from diverse disciplines and all other interested people, we show how multimedia containers can
support the visualization and sonification of scientific data. The use of a container format allows explorative
multimodal data analyses with any multimedia player as well as streaming the data via the Internet. We prototyped
this ap
proach on two datasets, both with visualization of gaze data and one with additional sonification
of EEG data. In a user study, we asked expert and non-expert users about their experience during an explorative
investigation of the data. Based on their statements, our prototype implementation, and the datasets,
we discuss the benefit of storing multimodal data, including the corresponding videos or images, in a single
multimedia container. In conclusion, we summarize what is necessary for having multimedia containers as a
standard for storing multimodal data and give an outlook on how artificial networks can be trained on such
standardized containers.
(More)