Cross-Stage Class-Specific Attention for Image Semantic Segmentation

Recent backbones built on transformers capture the context within a significantly larger area than CNN, and greatly improve the performance on semantic segmentation. However, the fact, that the decoder utilizes features from different stages in the shallow layers, indicates that local context is still important. Instead of simply incorporating features from different stages, we propose a cross-stage class-specific attention mainly for transformer-based backbones. Specifically, given a coarse prediction, we first employ the final stage features to aggregate a class center within the whole image. Then high-resolution features from the earlier stage are used as queries to absorb the semantics from class centers. To eliminate the irrelevant classes within a local area, we build the context for each query position according to the classification score from coarse prediction, and remove the redundant classes. So only relevant classes provide keys and values in attention and participate the value routing. We validate the proposed scheme on different datasets including ADE20K, Pascal Context and COCO-Stuff, showing that the proposed model improves the performance compared with other works.