Deformable and Structural Representative Network for Remote Sensing Image Captioning

Jaya Sharm, Peketi Divya, C. Vishnu, C. Reddy, B. Sekhar, C. Mohan

2023

Abstract

Remote sensing image captioning has greater significance in image understanding that generates textual description of aerial images automatically. Majority of the existing architectures work within the framework of encoder-decoder structure. However, it is noted that the existing encoder-decoder based methods for remote sensing image captioning avoid fine-grained structural representation of objects and lack deep encoding representation of an image. In this paper, we propose a novel structural representative network for capturing fine-grained structures of remote sensing imagery to produce fine grained captions. Initially, a deformable network has been incorporated on intermediate layers of convolutional neural network to take out spatially invariant features from an image. Secondly, a contextual network is incorporated in the last layers of the proposed network for producing multi-level contextual features. In order to extract dense contextual features, an attention mechanism is accomplished in contextual networks. Thus, the holistic representations of aerial images are obtained through a structural representative network by combining spatial and contextual features. Further, features from the structural representative network are provided to multi-level decoders for generating spatially semantic meaningful captions. The textual descriptions obtained due to our proposed approach is demonstrated on two standard datasets, namely, the Sydney-Captions dataset and the UCM-Captions dataset. The comparative analysis is made with recently proposed approaches to exhibit the performance of the proposed approach and hence argue that the proposed approach is more suitable for remote sensing image captioning tasks.

Paper Citation

in Harvard Style

Sharm J., Divya P., Vishnu C., Reddy C., Sekhar B. and Mohan C. (2023). Deformable and Structural Representative Network for Remote Sensing Image Captioning. In Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 4: VISAPP; ISBN 978-989-758-634-7, SciTePress, pages 56-64. DOI: 10.5220/0011625900003417

in Bibtex Style

@conference{visapp23,
author={Jaya Sharm and Peketi Divya and C. Vishnu and C. Reddy and B. Sekhar and C. Mohan},
title={Deformable and Structural Representative Network for Remote Sensing Image Captioning},
booktitle={Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 4: VISAPP},
year={2023},
pages={56-64},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011625900003417},
isbn={978-989-758-634-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 4: VISAPP
TI - Deformable and Structural Representative Network for Remote Sensing Image Captioning
SN - 978-989-758-634-7
AU - Sharm J.
AU - Divya P.
AU - Vishnu C.
AU - Reddy C.
AU - Sekhar B.
AU - Mohan C.
PY - 2023
SP - 56
EP - 64
DO - 10.5220/0011625900003417
PB - SciTePress