TDFNet: twice decoding V-Mamba-CNN Fusion features for building extraction

Building extraction from remote sensing imagery is vital for various human activities. But it is challenging due to diverse building appearances and complex backgrounds. Research shows the importance of both global context and spatial details for accurate building extraction. Therefore, methods integrating convolutional neural networks (CNNs) and visual transformers (ViTs) are popular nowadays. However, current methods combining these two methods inadequately merge their features and only perform decoding once, leading to issues like unclear boundaries, internal voids, and susceptibility to non-building elements in complex scenarios with low inter-class and high intra-class variability. To address these issues, this paper introduces a novel extraction method called TDFNet. We first replace ViT with V-Mamba, which has linear complexity, and combine it with CNN for feature extraction. A bidirectional fusion module (BFM) is then designed to comprehensively integrate spatial details and global information, thereby enabling accurate identification of boundaries between adjacent buildings, and maintaining the structural integrity of buildings to avoid internal holes. During the decoding process, we propose an Encoder-Decoder Fusion Module (EDFM) to initially merge features from different stages of the encoder and decoder, thereby diminishing the model’s susceptibility to non-building elements with features similar to those of buildings, and consequently reducing the incidence of erroneous extractions. Subsequently, a twice decoding strategy is implemented to enhance the learning of multi-scale features significantly, thereby mitigating the impact of tree occlusions and shadows. Our method yields the state-of-the-art (SOTA) performance on three public building datasets.