Skip to main navigation Skip to search Skip to main content

A Transformer-Based Decoder for Semantic Segmentation with Multi-level Context Mining

  • Bowen Shi
  • , Dongsheng Jiang
  • , Xiaopeng Zhang
  • , Han Li
  • , Wenrui Dai
  • , Junni Zou
  • , Hongkai Xiong
  • , Qi Tian*
  • *Corresponding author for this work
  • Shanghai Jiao Tong University
  • Huawei Technologies Co., Ltd.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Transformers have recently shown superior performance than CNN on semantic segmentation. However, previous works mostly focus on the deliberate design of the encoder, while seldom considering the decoder part. In this paper, we find that a light weighted decoder counts for segmentation, and propose a pure transformer-based segmentation decoder, named SegDeformer, to seamlessly incorporate into current varied transformer-based encoders. The highlight is that SegDeformer is able to conveniently utilize the tokenized input and the attention mechanism of the transformer for effective context mining. This is achieved by two key component designs, i.e., the internal and external context mining modules. The former is equipped with internal attention within an image to better capture global-local context, while the latter introduces external tokens from other images to enhance current representation. To enable SegDeformer in a scalable way, we further provide performance/efficiency optimization modules for flexible deployment. Experiments on widely used benchmarks ADE20K, COCO-Stuff and Cityscapes and different transformer encoders (e.g., ViT, MiT and Swin) demonstrate that SegDeformer can bring consistent performance gains. Code is available at https://github.com/lygsbw/segdeformer.

Original languageEnglish
Title of host publicationComputer Vision – ECCV 2022 - 17th European Conference, Proceedings
EditorsShai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
PublisherSpringer Science and Business Media Deutschland GmbH
Pages624-639
Number of pages16
ISBN (Print)9783031198144
DOIs
StatePublished - 2022
Externally publishedYes
Event17th European Conference on Computer Vision, ECCV 2022 - Tel Aviv, Israel
Duration: 23 Oct 2022 → 27 Oct 2022

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13688 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference17th European Conference on Computer Vision, ECCV 2022
Country/TerritoryIsrael
CityTel Aviv
Period23/10/22 → 27/10/22

Fingerprint

Dive into the research topics of 'A Transformer-Based Decoder for Semantic Segmentation with Multi-level Context Mining'. Together they form a unique fingerprint.

Cite this