Resources and End-to-End Neural Network Models for Arabic Image Captioning

Obeida ElJundi, Mohamad Dhaybi, Kotaiba Mokadam, Hazem Hajj, Daniel Asmar

2020

Abstract

Image Captioning (IC) is the process of automatically augmenting an image with semantically-laden descriptive text. While English IC has made remarkable strides forward in the past decade, very little work exists on IC for other languages. One possible solution to this problem is to boostrap off of existing English IC systems for image understanding, and then translate the outcome to the required language. Unfortunately, as this paper will show, translated IC is lacking due to the error accumulation of the two tasks; IC and translation. In this paper, we address the problem of image captioning in Arabic. We propose an end-to-end model that directly transcribes images into Arabic text. Due to the lack of Arabic resources, we develop an annotated dataset for Arabic image captioning (AIC). We also develop a base model for AIC that relies on text translation from English image captions. The two models are evaluated with the new dataset, and the results show the superiority of our end-to-end model.

Paper Citation

in Harvard Style

ElJundi O., Dhaybi M., Mokadam K., Hajj H. and Asmar D. (2020). Resources and End-to-End Neural Network Models for Arabic Image Captioning. In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 5: VISAPP; ISBN 978-989-758-402-2, SciTePress, pages 233-241. DOI: 10.5220/0008881202330241

in Bibtex Style

@conference{visapp20,
author={Obeida ElJundi and Mohamad Dhaybi and Kotaiba Mokadam and Hazem Hajj and Daniel Asmar},
title={Resources and End-to-End Neural Network Models for Arabic Image Captioning},
booktitle={Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 5: VISAPP},
year={2020},
pages={233-241},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0008881202330241},
isbn={978-989-758-402-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 5: VISAPP
TI - Resources and End-to-End Neural Network Models for Arabic Image Captioning
SN - 978-989-758-402-2
AU - ElJundi O.
AU - Dhaybi M.
AU - Mokadam K.
AU - Hajj H.
AU - Asmar D.
PY - 2020
SP - 233
EP - 241
DO - 10.5220/0008881202330241
PB - SciTePress