Skip to main navigation Skip to search Skip to main content

VioLET: Vision-Language Efficient Tuning with Collaborative Multi-modal Gradients

  • Yaoming Wang
  • , Yuchen Liu
  • , Xiaopeng Zhang
  • , Jin Li
  • , Bowen Shi
  • , Chenglin Li
  • , Wenrui Dai
  • , Hongkai Xiong
  • , Qi Tian*
  • *Corresponding author for this work
  • Shanghai Jiao Tong University
  • Huawei Technologies Co., Ltd.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Parameter-Efficient Tuning (PET) has emerged as a leading advancement in both Natural Language Processing and Computer Vision, enabling efficient accommodation of downstream tasks without costly fine-tuning. However, most existing PET approaches are limited to uni-modal tuning, even for vision-language models like CLIP. We investigate this limitation and demonstrate that simultaneous tuning of the two modalities in such models leads to multi-modal forgetting and catastrophic performance degradation, particularly when generalizing to new classes. To address this issue, we propose a novel PET approach called VioLET (Vision Language Efficient Tuning) that utilizes collaborative multi-modal gradients to unlock the full potential of both modalities. Specifically, we incorporate an additional visual encoder without learnable parameters and use these two visual encoders to compute the gradients of the context parameters separately. When conflicts arise, we replace the original gradient with an orthogonal gradient. Extensive experiments are conducted on few-shot recognition and unseen class generalization tasks using ResNet-50 or ViT/B-16 as the backbone. VioLET consistently outperforms several state-of-the-art methods on 11 datasets, showcasing its superiority over existing PET approaches. The code is available at https://github.com/Wang-Yaoming/VioLET.

Original languageEnglish
Title of host publicationMM 2023 - Proceedings of the 31st ACM International Conference on Multimedia
PublisherAssociation for Computing Machinery, Inc
Pages4595-4605
Number of pages11
ISBN (Electronic)9798400701085
DOIs
StatePublished - 27 Oct 2023
Externally publishedYes
Event31st ACM International Conference on Multimedia, MM 2023 - Ottawa, Canada
Duration: 29 Oct 2023 → 3 Nov 2023

Publication series

NameMM 2023 - Proceedings of the 31st ACM International Conference on Multimedia

Conference

Conference31st ACM International Conference on Multimedia, MM 2023
Country/TerritoryCanada
CityOttawa
Period29/10/23 → 3/11/23

Keywords

  • few-shot recognition
  • multi-modal
  • parameter efficient tuning
  • prompt learning
  • vision language

Fingerprint

Dive into the research topics of 'VioLET: Vision-Language Efficient Tuning with Collaborative Multi-modal Gradients'. Together they form a unique fingerprint.

Cite this