A Comprehensive Study of Vision Transformers on Dense Prediction Tasks

Kishaan Jeeveswaran, Senthilkumar Kathiresan, Arnav Varma, Omar Magdy, Bahram Zonooz, Elahe Arani

2022

Abstract

Convolutional Neural Networks (CNNs), architectures consisting of convolutional layers, have been the standard choice in vision tasks. Recent studies have shown that Vision Transformers (VTs), architectures based on self-attention modules, achieve comparable performance in challenging tasks such as object detection and semantic segmentation. However, the image processing mechanism of VTs is different from that of conventional CNNs. This poses several questions about their generalizability, robustness, reliability, and texture bias when used to extract features for complex tasks. To address these questions, we study and compare VT and CNN architectures as a feature extractor in object detection and semantic segmentation. Our extensive empirical results show that the features generated by VTs are more robust to distribution shifts, natural corruptions, and adversarial attacks in both tasks, whereas CNNs perform better at higher image resolutions in object detection. Furthermore, our results demonstrate that VTs in dense prediction tasks produce more reliable and less texture biased predictions.

Paper Citation

in Harvard Style

Jeeveswaran K., Kathiresan S., Varma A., Magdy O., Zonooz B. and Arani E. (2022). A Comprehensive Study of Vision Transformers on Dense Prediction Tasks. In Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 4: VISAPP; ISBN 978-989-758-555-5, SciTePress, pages 213-223. DOI: 10.5220/0010917800003124

in Bibtex Style

@conference{visapp22,
author={Kishaan Jeeveswaran and Senthilkumar Kathiresan and Arnav Varma and Omar Magdy and Bahram Zonooz and Elahe Arani},
title={A Comprehensive Study of Vision Transformers on Dense Prediction Tasks},
booktitle={Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 4: VISAPP},
year={2022},
pages={213-223},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010917800003124},
isbn={978-989-758-555-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 4: VISAPP
TI - A Comprehensive Study of Vision Transformers on Dense Prediction Tasks
SN - 978-989-758-555-5
AU - Jeeveswaran K.
AU - Kathiresan S.
AU - Varma A.
AU - Magdy O.
AU - Zonooz B.
AU - Arani E.
PY - 2022
SP - 213
EP - 223
DO - 10.5220/0010917800003124
PB - SciTePress