Factor Space and Spectrum for Medical Hyperspectral Image Segmentation

Medical Hyperspectral Imaging (MHSI) brings opportunities for computational pathology and precision medicine. Since MHSI is a 3D hypercube, building a 3D segmentation network is the most intuitive way for MHSI segmentation. But, high spatiospectral dimensions make it difficult to perform efficient and effective segmentation. In this study, in light of information correlation in MHSIs, we present a computationally efficient, plug-and-play space and spectrum factorization strategy based on 2D architectures. Drawing inspiration from the low-rank prior of MHSIs, we propose spectral matrix decomposition and low-rank decomposition modules for removing redundant spatiospectral information. By plugging our dual-stream strategy into 2D backbones, we can achieve state-of-the-art MHSI segmentation performances with 3–13 times faster compared with existing 3D networks in terms of inference speed. Experiments show our strategy leads to remarkable performance gains in different 2D architectures, reporting an improvement up to $$7.7\%$$ compared with its 2D counterpart in terms of DSC on a public Multi-Dimensional Choledoch dataset. Code is publicly available at https://github.com/boxiangyun/Dual-Stream-MHSI.