跳到主要导航 跳到搜索 跳到主要内容

Adapting Shortcut with Normalizing Flow: An Efficient Tuning Framework for Visual Recognition

  • Yaoming Wang
  • , Bowen Shi
  • , Xiaopeng Zhang
  • , Jin Li
  • , Yuchen Liu
  • , Wenrui Dai
  • , Chenglin Li
  • , Hongkai Xiong
  • , Qi Tian*
  • *此作品的通讯作者
  • Shanghai Jiao Tong University
  • Huawei Cloud Computing Technologies

科研成果: 书/报告/会议事项章节 › 会议稿件 › 同行评审

摘要

Pretraining followed by fine-tuning has proven to be effective in visual recognition tasks. However, fine-tuning all parameters can be computationally expensive, particularly for large-scale models. To mitigate the computational and storage demands, recent research has explored Parameter-Efficient Fine-Tuning (PEFT), which focuses on tuning a minimal number of parameters for efficient adaptation. Existing methods, however, fail to analyze the impact of the additional parameters on the model, resulting in an unclear and suboptimal tuning process. In this paper, we introduce a novel and effective PEFT paradigm, named SNF (Shortcut adaptation via Normalization Flow), which utilizes normalizing flows to adjust the shortcut layers. We highlight that layers without Lipschitz constraints can lead to error propagation when adapting to downstream datasets. Since modifying the over-parameterized residual connections in these layers is expensive, we focus on adjusting the cheap yet crucial shortcuts. Moreover, learning new information with few parameters in PEFT can be challenging, and information loss can result in label information degradation. To address this issue, we propose an information-preserving normalizing flow. Experimental results demonstrate the effectiveness of SNF. Specifically, with only 0.036M parameters, SNF surpasses previous approaches on both the FGVC and VTAB-1k benchmarks using ViT/B-16 as the backbone. The code is available at https://github.com/Wang-Yaoming/SNF

源语言英语
主期刊名Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
出版商IEEE Computer Society
页15965-15974
页数10
ISBN(电子版)9798350301298
DOI
出版状态已出版 - 2023
已对外发布是
活动2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 - Vancouver, 加拿大
期限: 18 6月 2023 → 22 6月 2023

丛书

姓名Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
卷2023-June
ISSN(印刷版)1063-6919

会议

会议2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
国家/地区加拿大
市Vancouver
时期18/06/23 → 22/06/23

学术指纹

探究 'Adapting Shortcut with Normalizing Flow: An Efficient Tuning Framework for Visual Recognition' 的科研主题。它们共同构成独一无二的学术指纹。

引用此