Dynamic Hierarchical Token Merging for Vision Transformers

Karim Haroun, Karim Haroun, Thibault Allenet, Karim Ben Chehida, Jean Martinet

2025

Abstract

Vision Transformers (ViTs) have achieved impressive results in computer vision, excelling in tasks such as image classification, segmentation, and object detection. However, their quadratic complexity O(N2), where N is the token sequence length, poses challenges when deployed on resource-limited devices. To address this issue, dynamic token merging has emerged as an effective strategy, progressively reducing the token count during inference to achieve computational savings. Some strategies consider all tokens in the sequence as merging candidates, without focusing on spatially close tokens. Other strategies either limit token merging to a local window, or constrains it to pairs of adjacent tokens, thus not capturing more complex feature relationships. In this paper, we propose Dynamic Hierarchical Token Merging (DHTM), a novel token merging approach, where we advocate that spatially close tokens share more information than distant tokens and consider all pairs of spatially close candidates instead of imposing fixed windows. Besides, our approach draws on the principles of Hierarchical Agglomerative Clustering (HAC), where we iteratively merge tokens in each layer, fusing a fixed number of selected neighbor token pairs based on their similarity. Our proposed approach is off-the-shelf, i.e., it does not require additional training. We evaluate our approach on the ImageNet-1K dataset for classification, achieving substantial computational savings while minimizing accuracy reduction, surpassing existing token merging methods.

Paper Citation

in Harvard Style

Haroun K., Allenet T., Ben Chehida K. and Martinet J. (2025). Dynamic Hierarchical Token Merging for Vision Transformers. In Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP; ISBN 978-989-758-728-3, SciTePress, pages 677-684. DOI: 10.5220/0013284100003912

in Bibtex Style

@conference{visapp25,
author={Karim Haroun and Thibault Allenet and Karim Ben Chehida and Jean Martinet},
title={Dynamic Hierarchical Token Merging for Vision Transformers},
booktitle={Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP},
year={2025},
pages={677-684},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013284100003912},
isbn={978-989-758-728-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP
TI - Dynamic Hierarchical Token Merging for Vision Transformers
SN - 978-989-758-728-3
AU - Haroun K.
AU - Allenet T.
AU - Ben Chehida K.
AU - Martinet J.
PY - 2025
SP - 677
EP - 684
DO - 10.5220/0013284100003912
PB - SciTePress