Multimodal Sentiment Analysis: A Multitask Learning Approach

Mathieu Fortin, Brahim Chaib-Draa

2019

Abstract

Multimodal sentiment analysis has recently received an increasing interest. However, most methods have considered that text and image modalities are always available at test time. This assumption is often violated in real environments (e.g. social media) since users do not always publish a text with an image. In this paper we propose a method based on a multitask framework to combine multimodal information when it is available, while being able to handle the cases where a modality is missing. Our model contains one classifier for analyzing the text, another for analyzing the image, and another performing the prediction by fusing both modalities. In addition to offer a solution to the problem of a missing modality, our experiments show that this multitask framework improves generalization by acting as a regularization mechanism. We also demonstrate that the model can handle a missing modality at training time, thus being able to be trained with image-only and text-only examples.

Paper Citation

in Harvard Style

Fortin M. and Chaib-Draa B. (2019). Multimodal Sentiment Analysis: A Multitask Learning Approach.In Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-758-351-3, pages 368-376. DOI: 10.5220/0007313503680376

in Bibtex Style

@conference{icpram19,
author={Mathieu Fortin and Brahim Chaib-Draa},
title={Multimodal Sentiment Analysis: A Multitask Learning Approach},
booktitle={Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2019},
pages={368-376},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0007313503680376},
isbn={978-989-758-351-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - Multimodal Sentiment Analysis: A Multitask Learning Approach
SN - 978-989-758-351-3
AU - Fortin M.
AU - Chaib-Draa B.
PY - 2019
SP - 368
EP - 376
DO - 10.5220/0007313503680376