跳到主要导航 跳到搜索 跳到主要内容

TEXT4SEG: REIMAGINING IMAGE SEGMENTATION AS TEXT GENERATION

  • Mengcheng Lan
  • , Chaofeng Chen
  • , Yue Zhou
  • , Jiaxing Xu
  • , Yiping Ke
  • , Xinjiang Wang
  • , Litong Feng*
  • , Wayne Zhang
  • *此作品的通讯作者
  • Nanyang Technological University
  • SenseTime Group Limited

科研成果: 书/报告/会议事项章节 › 会议稿件 › 同行评审

摘要

Multimodal Large Language Models (MLLMs) have shown exceptional capabilities in vision-language tasks; however, effectively integrating image segmentation into these models remains a significant challenge. In this paper, we introduce Text4Seg, a novel text-as-mask paradigm that casts image segmentation as a text generation problem, eliminating the need for additional decoders and significantly simplifying the segmentation process. Our key innovation is semantic descriptors, a new textual representation of segmentation masks where each image patch is mapped to its corresponding text label. This unified representation allows seamless integration into the auto-regressive training pipeline of MLLMs for easier optimization. We demonstrate that representing an image with 16 × 16 semantic descriptors yields competitive segmentation performance. To enhance efficiency, we introduce the Row-wise Run-Length Encoding (R-RLE), which compresses redundant text sequences, reducing the length of semantic descriptors by 74% and accelerating inference by 3×, without compromising performance. Extensive experiments across various vision tasks, such as referring expression segmentation and comprehension, show that Text4Seg achieves state-of-the-art performance on multiple datasets by fine-tuning different MLLM backbones. Our approach provides an efficient, scalable solution for vision-centric tasks within the MLLM framework. https://github.com/mc-lan/Text4Seg.

源语言英语
主期刊名13th International Conference on Learning Representations, ICLR 2025
出版商International Conference on Learning Representations, ICLR
页89206-89233
页数28
ISBN(电子版)9798331320850
出版状态已出版 - 2025
已对外发布是
活动13th International Conference on Learning Representations, ICLR 2025 - Singapore, 新加坡
期限: 24 4月 2025 → 28 4月 2025

丛书

姓名13th International Conference on Learning Representations, ICLR 2025

会议

会议13th International Conference on Learning Representations, ICLR 2025
国家/地区新加坡
市Singapore
时期24/04/25 → 28/04/25

学术指纹

探究 'TEXT4SEG: REIMAGINING IMAGE SEGMENTATION AS TEXT GENERATION' 的科研主题。它们共同构成独一无二的学术指纹。

引用此