When Medical Imaging Met Self-Attention: A Love Story That Didn’t Quite Work out

Tristan Piater, Niklas Penzel, Gideon Stein, Joachim Denzler

2024

Abstract

A substantial body of research has focused on developing systems that assist medical professionals during labor-intensive early screening processes, many based on convolutional deep-learning architectures. Recently, multiple studies explored the application of so-called self-attention mechanisms in the vision domain. These studies often report empirical improvements over fully convolutional approaches on various datasets and tasks. To evaluate this trend for medical imaging, we extend two widely adopted convolutional architectures with different self-attention variants on two different medical datasets. With this, we aim to specifically evaluate the possible advantages of additional self-attention. We compare our models with similarly sized convolutional and attention-based baselines and evaluate performance gains statistically. Additionally, we investigate how including such layers changes the features learned by these models during the training. Following a hyperparameter search, and contrary to our expectations, we observe no significant improvement in balanced accuracy over fully convolutional models. We also find that important features, such as dermoscopic structures in skin lesion images, are still not learned by employing self-attention. Finally, analyzing local explanations, we confirm biased feature usage. We conclude that merely incorporating attention is insufficient to surpass the performance of existing fully convolutional methods.

Paper Citation

in Harvard Style

Piater T., Penzel N., Stein G. and Denzler J. (2024). When Medical Imaging Met Self-Attention: A Love Story That Didn’t Quite Work out. In Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP; ISBN 978-989-758-679-8, SciTePress, pages 149-158. DOI: 10.5220/0012382600003660

in Bibtex Style

@conference{visapp24,
author={Tristan Piater and Niklas Penzel and Gideon Stein and Joachim Denzler},
title={When Medical Imaging Met Self-Attention: A Love Story That Didn’t Quite Work out},
booktitle={Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP},
year={2024},
pages={149-158},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012382600003660},
isbn={978-989-758-679-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP
TI - When Medical Imaging Met Self-Attention: A Love Story That Didn’t Quite Work out
SN - 978-989-758-679-8
AU - Piater T.
AU - Penzel N.
AU - Stein G.
AU - Denzler J.
PY - 2024
SP - 149
EP - 158
DO - 10.5220/0012382600003660
PB - SciTePress