TAJ-Net: a two-stage clustered cell segmentation network with adaptive joint learning of spatial and spectral information

Pulmonary adenocarcinoma is the primary cause of cancer-related death worldwide and pathological diagnosis is the “golden standard” based on the regional distribution of cells. Thus, regional cell segmentation is a key step while it is challenging due to the following reasons: 1) It is hard for pure semantic and instance segmentation methods to obtain a high-quality regional cell segmentation result; 2) Since the spatial appearances of pulmonary cells are very similar which even confuse pathologists, annotation errors are usually inevitable. Considering these challenges, we propose a two-stage 3D adaptive joint training framework (TAJ-Net) to segment-then-classify cells with extra spectral information as the supplementary information of spatial information. Firstly, we propose to leverage a few-shot method with limited data for cell mask acquisition to avoid the disturbance of cluttered backgrounds. Secondly, we introduce an adaptive joint training strategy to remove noisy samples through two 3D networks and one 1D network for cell type classification rather than segmentation. Subsequently, we propose a patch mapping method to map classification results to the original images to obtain regional segmentation results. In order to verify the effectiveness of TAJ-Net, we build two 3D hyperspectral datasets, i.e., pulmonary adenocarcinoma (3,660 images) and thyroid carcinoma (4623 images) with 40 bands. The first dataset will be released for further research. Experiments show that TAJ-Net achieves much better performance in clustered cell segmentation, and it can regionally segment different kinds of cells with high overlap and blurred edges, which is a difficult task for the state-of-the-art methods. Compared to 2D models, the hyperspectral image-based 3D model reports a significant improvement of up to 11.5% in terms of the Dice similarity coefficient in the pulmonary adenocarcinoma dataset.