跳到主要导航 跳到搜索 跳到主要内容

MVP-SEG: Multi-view Prompt Learning for Open-Vocabulary Semantic Segmentation

  • Jie Guo
  • , Qimeng Wang
  • , Yan Gao
  • , Xiaolong Jiang
  • , Shaohui Lin
  • , Baochang Zhang*
  • *此作品的通讯作者
  • Beihang University
  • Xiaohongshu
  • KLATASDS-MOE
  • Nanchang Institute of Technology

科研成果: 书/报告/会议事项章节 › 会议稿件 › 同行评审

摘要

CLIP (Contrastive Language-Image Pretraining) is well developed for open-vocabulary zero-shot image-level recognition, while its applications in pixel-level tasks are less investigated, where most efforts directly adopt CLIP features without deliberative adaptations. In this work, we first demonstrate the necessity of image-pixel CLIP feature adaption, then provide Multi-View Prompt learning (MVP-SEG) as an effective solution to achieve image-pixel adaptation and to solve open-vocabulary semantic segmentation. Concretely, MVP-SEG deliberately learns multiple prompts trained by our Orthogonal Constraint Loss (OCLoss), by which each prompt is supervised to exploit CLIP feature on different object parts, and collaborative segmentation masks generated by all prompts promote better segmentation. Moreover, MVP-SEG introduces Global Prompt Refining (GPR) to further eliminate class-wise segmentation noise. Experiments show that the multi-view prompts learned from seen categories have strong generalization to unseen categories, and MVP-SEG+ which combines the knowledge transfer stage significantly outperforms previous methods on several benchmarks. Moreover, qualitative results justify that MVP-SEG does lead to better focus on different local parts.

源语言英语
主期刊名Pattern Recognition and Computer Vision - 6th Chinese Conference, PRCV 2023, Proceedings
编辑Qingshan Liu, Hanzi Wang, Rongrong Ji, Zhanyu Ma, Weishi Zheng, Hongbin Zha, Xilin Chen, Liang Wang
出版商Springer Science and Business Media Deutschland GmbH
页158-171
页数14
ISBN(印刷版)9789819985548
DOI
出版状态已出版 - 2024
活动6th Chinese Conference on Pattern Recognition and Computer Vision, PRCV 2023 - Xiamen, 中国
期限: 13 10月 2023 → 15 10月 2023

丛书

姓名Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
卷14436 LNCS
ISSN(印刷版)0302-9743
ISSN(电子版)1611-3349

会议

会议6th Chinese Conference on Pattern Recognition and Computer Vision, PRCV 2023
国家/地区中国
市Xiamen
时期13/10/23 → 15/10/23

学术指纹

探究 'MVP-SEG: Multi-view Prompt Learning for Open-Vocabulary Semantic Segmentation' 的科研主题。它们共同构成独一无二的学术指纹。

引用此