TY - GEN
T1 - A Multiplier-Free Similarity Metric for Token Merging in Vision Transformers
AU - Gui, Yupeng
AU - Tang, Ruiqi
AU - Peng, Shichen
AU - He, Han
AU - Zhou, Sheng
AU - Huang, Leilei
AU - Yan, Xiao
AU - Fan, Yibo
N1 - Publisher Copyright:
© 2026 IEEE.
PY - 2026
Y1 - 2026
N2 - Token Merging (TM) has become an effective and widely adopted strategy for accelerating Vision Transformers (ViTs), reducing computational cost by merging similar tokens without requiring model retraining. However, the metric used to compute this pairwise similarity suffers from an accuracy-cost dilemma. Cosine similarity, despite being the de facto standard, is computationally expensive for hardware implementations, while existing lightweight metrics yield only moderate performance. To resolve this, we introduce the Transform-Domain L1 Distance (TDL), a multiplier-free similarity metric with competitive model accuracy. This paper first proposes an automated search strategy using Simulated Annealing (SA) to discover optimal transform kernels. With these SA-found kernels, TDL allows the host model to sustain or surpass task accuracy compared with cosine similarity. Then, through comprehensive comparison, we identify the Hadamard kernel as a highly effective universal transform kernel. By exploiting the Fast Hadamard Transform (FHT), TDL with Hadamard kernel delivers exceptional hardware efficiency while maintaining almost the same accuracy with (and often exceeding) cosine similarity, establishing a new Pareto-optimal solution for hardware-efficient TM design.
AB - Token Merging (TM) has become an effective and widely adopted strategy for accelerating Vision Transformers (ViTs), reducing computational cost by merging similar tokens without requiring model retraining. However, the metric used to compute this pairwise similarity suffers from an accuracy-cost dilemma. Cosine similarity, despite being the de facto standard, is computationally expensive for hardware implementations, while existing lightweight metrics yield only moderate performance. To resolve this, we introduce the Transform-Domain L1 Distance (TDL), a multiplier-free similarity metric with competitive model accuracy. This paper first proposes an automated search strategy using Simulated Annealing (SA) to discover optimal transform kernels. With these SA-found kernels, TDL allows the host model to sustain or surpass task accuracy compared with cosine similarity. Then, through comprehensive comparison, we identify the Hadamard kernel as a highly effective universal transform kernel. By exploiting the Fast Hadamard Transform (FHT), TDL with Hadamard kernel delivers exceptional hardware efficiency while maintaining almost the same accuracy with (and often exceeding) cosine similarity, establishing a new Pareto-optimal solution for hardware-efficient TM design.
KW - Hardware Accelerator
KW - Similarity Metric
KW - Token Merging
KW - Vision Transformer
UR - https://www.scopus.com/pages/publications/105043510843
U2 - 10.1109/ISCAS66217.2026.11562161
DO - 10.1109/ISCAS66217.2026.11562161
M3 - 会议稿件
AN - SCOPUS:105043510843
T3 - Proceedings - IEEE International Symposium on Circuits and Systems
SP - 1839
EP - 1843
BT - ISCAS 2026 - 2026 IEEE International Symposium on Circuits and Systems
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2026 IEEE International Symposium on Circuits and Systems, ISCAS 2026
Y2 - 24 May 2026 through 27 May 2026
ER -