loading
Papers Papers/2022 Papers Papers/2022

Research.Publish.Connect.

Paper

Authors: Tomas Hrycej ; Bernhard Bermeitinger and Siegfried Handschuh

Affiliation: Institute of Computer Science, University of St. Gallen (HSG), St. Gallen, Switzerland

Keyword(s): Computer Vision, Transformer, Attention, Comparison.

Abstract: Determining an appropriate number of attention heads on one hand and the number of transformer-encoders, on the other hand, is an important choice for Computer Vision (CV) tasks using the Transformer architecture. Computing experiments confirmed the expectation that the total number of parameters has to satisfy the condition of overdetermination (i.e., number of constraints significantly exceeding the number of parameters). Then, good generalization performance can be expected. This sets the boundaries within which the number of heads and the number of transformers can be chosen. If the role of context in images to be classified can be assumed to be small, it is favorable to use multiple transformers with a low number of heads (such as one or two). In classifying objects whose class may heavily depend on the context within the image (i.e., the meaning of a patch being dependent on other patches), the number of heads is equally important as that of transformers.

CC BY-NC-ND 4.0

Sign In Guest: Register as new SciTePress user now for free.

Sign In SciTePress user: please login.

PDF ImageMy Papers

You are not signed in, therefore limits apply to your IP address 3.145.163.138

In the current month:
Recent papers: 100 available of 100 total
2+ years older papers: 200 available of 200 total

Paper citation in several formats:
Hrycej, T.; Bermeitinger, B. and Handschuh, S. (2022). Number of Attention Heads vs. Number of Transformer-encoders in Computer Vision. In Proceedings of the 14th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2022) - KDIR; ISBN 978-989-758-614-9; ISSN 2184-3228, SciTePress, pages 315-321. DOI: 10.5220/0011578000003335

@conference{kdir22,
author={Tomas Hrycej. and Bernhard Bermeitinger. and Siegfried Handschuh.},
title={Number of Attention Heads vs. Number of Transformer-encoders in Computer Vision},
booktitle={Proceedings of the 14th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2022) - KDIR},
year={2022},
pages={315-321},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011578000003335},
isbn={978-989-758-614-9},
issn={2184-3228},
}

TY - CONF

JO - Proceedings of the 14th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2022) - KDIR
TI - Number of Attention Heads vs. Number of Transformer-encoders in Computer Vision
SN - 978-989-758-614-9
IS - 2184-3228
AU - Hrycej, T.
AU - Bermeitinger, B.
AU - Handschuh, S.
PY - 2022
SP - 315
EP - 321
DO - 10.5220/0011578000003335
PB - SciTePress

- Science and Technology Publications, Lda.
RESOURCES

Proceedings

Papers

Authors

Ontology

CONTACTS

Science and Technology Publications, Lda
Avenida de S. Francisco Xavier, Lote 7 Cv. C,
2900-616 Setúbal, Portugal.

Phone: +351 265 520 185 (National fixed network call)
Fax: +351 265 520 186
Email: info@scitepress.org

EXTERNAL LINKS

PRIMORIS

INSTICC

SCITEVENTS

CROSSREF

PROCEEDINGS SUBMITTED FOR INDEXATION BY:

dblp

Ei Compendex

SCOPUS

Semantic Scholar

Google Scholar

Microsoft Academic