跳到主要导航 跳到搜索 跳到主要内容

A Multiplier-Free Similarity Metric for Token Merging in Vision Transformers

  • Yupeng Gui
  • , Ruiqi Tang
  • , Shichen Peng
  • , Han He
  • , Sheng Zhou
  • , Leilei Huang
  • , Xiao Yan*
  • , Yibo Fan
  • *此作品的通讯作者
  • Fudan University
  • Xi'an Institute of Posts and Telecommunications

科研成果: 书/报告/会议事项章节 › 会议稿件 › 同行评审

摘要

Token Merging (TM) has become an effective and widely adopted strategy for accelerating Vision Transformers (ViTs), reducing computational cost by merging similar tokens without requiring model retraining. However, the metric used to compute this pairwise similarity suffers from an accuracy-cost dilemma. Cosine similarity, despite being the de facto standard, is computationally expensive for hardware implementations, while existing lightweight metrics yield only moderate performance. To resolve this, we introduce the Transform-Domain L1 Distance (TDL), a multiplier-free similarity metric with competitive model accuracy. This paper first proposes an automated search strategy using Simulated Annealing (SA) to discover optimal transform kernels. With these SA-found kernels, TDL allows the host model to sustain or surpass task accuracy compared with cosine similarity. Then, through comprehensive comparison, we identify the Hadamard kernel as a highly effective universal transform kernel. By exploiting the Fast Hadamard Transform (FHT), TDL with Hadamard kernel delivers exceptional hardware efficiency while maintaining almost the same accuracy with (and often exceeding) cosine similarity, establishing a new Pareto-optimal solution for hardware-efficient TM design.

源语言英语
主期刊名ISCAS 2026 - 2026 IEEE International Symposium on Circuits and Systems
出版商Institute of Electrical and Electronics Engineers Inc.
页1839-1843
页数5
ISBN(电子版)9798331577698
DOI
出版状态已出版 - 2026
活动2026 IEEE International Symposium on Circuits and Systems, ISCAS 2026 - Shanghai, 中国
期限: 24 5月 2026 → 27 5月 2026

丛书

姓名Proceedings - IEEE International Symposium on Circuits and Systems
ISSN(印刷版)0271-4310

会议

会议2026 IEEE International Symposium on Circuits and Systems, ISCAS 2026
国家/地区中国
市Shanghai
时期24/05/26 → 27/05/26

学术指纹

探究 'A Multiplier-Free Similarity Metric for Token Merging in Vision Transformers' 的科研主题。它们共同构成独一无二的学术指纹。

引用此