TL-MSE²-Net: Transfer learning based nested model for cerebrovascular segmentation with aneurysms

Cerebrovascular (i.e., cerebral vessel) segmentation is essential for diagnosing and treating brain diseases. Convolutional neural network models, such as U-Net, are commonly used for this purpose. Unfortunately, such models may not be entirely satisfactory in dealing with cerebrovascular segmentation with tumors due to the following issues: (1) Relatively small number of clinical datasets from patients obtained through different modalities such as computed tomography (CT) and magnetic resonance imaging (MRI), leading to inadequate training and lack of transferability in the modeling; (2) Insufficient feature extraction caused by less attention to both convolution sizes and cerebral vessel edges. Inspired by the existence of similar features on cerebral vessels between normal subjects and patients, we propose a transfer learning strategy based on a pre-trained nested model called TL-MSE²-Net. This model uses one of the publicly available datasets for cerebrovascular segmentation with aneurysms. To address issue (1), our transfer learning strategy leverages a pre-trained model that uses a large number of datasets from normal subjects, providing a potential solution to the lack of sufficient clinical datasets. To tackle issue (2), we structure the pre-trained model based on 3D U-Net, comprising three blocks: ResMul, DeRes, and REAM. The ResMul and DeRes blocks enhance feature extraction by utilizing multiple convolution sizes to capture multiscale features, and the REAM block increases the weight of the voxels on the edges of the given 3D volume. We evaluated the proposed model on one small private clinical dataset and two publicly available datasets. The experimental results demonstrated that our MSE²-Net framework achieved an average Dice score of 70.81 % and 89.08 % on the two publicly available datasets, outperforming other state-of-the-art methods. Ablation studies were also conducted to validate the effectiveness of each block. The proposed TL-MSE²-Net yielded better results than MSE²-Net on a small private clinical dataset, with increases of 5.52 %, 3.37 %, 6.71 %, and 0.85 % for the Dice score, sensitivity, Jaccard index, and precision, respectively.