Glioma grading based on 3D multimodal convolutional neural network and privileged learning

Brain tumors, especially high-grade gliomas, are one of the most lethal cancers for humankind today. Early and accurate diagnosis of tumor grading is the key for subsequent therapy and treatment. In the past, conventional computer-aided diagnosis relies on handcrafted features from magnetic resonance images (MRI), which are usually inaccurate and laborious. Recently, deep neural networks have been developed and applied for tumor segmentation and classification. However, most existing methods consider 3D MRI as a series of 2D images and use a simple modality fusion method via feature concatenation. In this paper, we propose an end-to-end 3-dimensional convolutional neural network (3D CNN) with gated multimodal unit (GMU) fusion to integrate the information both in three dimensions and in multiple modalities. Specifically, 3D convolutional kernels are directly applied to the whole MRI images, gathering the abnormalities in sagittal, axial and coronal directions. GMU with hidden states is proposed to fuse the information of multiple MRI modalities in both feature and decision level. Based on these, privilege information extracted by GMU fusion model is utilized to train a novel network called distilled-CNN, which significantly improves the performance of classification using single modality. Empirical studies on BRATS datasets corroborate the effectiveness of the proposed 3D CNN with GMU fusion and distilled-CNN to distinguish benign gliomas and malignant gliomas.