A novel inference paradigm based on multi-view prototypes for one-shot semantic segmentation

Abstract: One-shot semantic segmentation approaches aim to learn a meta-learning framework from seen classes with annotated samples, which can be applied in novel classes with only one annotated sample. However, most existing works still face the challenge of reduced generalization capability on novel classes due to two reasons: utilizing only foreground and background prototypes generated from support samples may lead to semantic bias from the model’s perspective, and negative support-query pairs may result in spatial inconsistency from the data’s perspective. To alleviate the semantic bias problem, we propose a multi-view prototype learning paradigm to reduce the appearance discrepancy between support and query images. In addition to the classical foreground and background prototypes, the multi-view prototypes include support outline view, query foreground view, seen class object view and natural background view prototypes. These proposed prototypes provide more refined semantic support information. To reduce the impact of negative samples, we propose a novel inference paradigm (n-iteration inference) for producing pseudo labels of novel classes as augmented support samples. These samples are then applied in the proposed multi-view prototype method for one-shot semantic segmentation. Experimental results show that we have achieved new state-of-the-art performance on the two standard datasets, PASCAL-5 ⁱ and COCO-20 ⁱ . Furthermore, we apply the inference paradigm to other classical works in order to enhance the performance of one-shot semantic segmentation. Our source code will be available on https://github.com/WHL182/MVPNet. Graphical Abstract: (left).[Figure not available: see fulltext.].