Automated Deep Learning-based Segmentation of the Dentate Nucleus Using Quantitative Susceptibility Mapping MRI

Purpose To develop a dentate nucleus (DN) segmentation tool using deep learning applied to brain MRI-based quantitative susceptibility mapping (QSM) images. Materials and Methods Brain QSM images from healthy controls and individuals with cerebellar ataxia or multiple sclerosis were collected from nine different datasets (2016-2023) worldwide for this retrospective study (ClinicalTrials.gov identifier: NCT04349514). Manual delineation of the DN was performed by experienced raters. Automated segmentation performance was evaluated against manual reference segmentations following training with several deep learning architectures. A two-step approach was used, consisting of a localization model followed by DN segmentation. Performance metrics included intraclass correlation coefficient (ICC), Dice score, and Pearson correlation coefficient. Results The training and testing datasets comprised 328 individuals (age range, 11-64 years; 171 female individuals), including 141 healthy individuals and 187 with cerebellar ataxia or multiple sclerosis. The manual tracing protocol produced reference standards with high intrarater (average ICC, 0.91) and interrater reliability (average ICC, 0.78). Initial deep learning architecture exploration indicated that the nnU-Net framework performed best. The two-step localization plus segmentation pipeline achieved a Dice score of 0.90 ± 0.03 (SD) and 0.89 ± 0.04 for left and right DN segmentation, respectively. In external testing, the proposed algorithm outperformed the current leading automated tool (mean Dice scores for left and right DN, 0.86 ± 0.04 vs 0.57 ± 0.22 [P < .001]; 0.84 ± 0.07 vs 0.58 ± 0.24 [P < .001]). The model demonstrated generalizability across datasets unseen during the training step, with automated segmentations showing high correlation with manual annotations (left DN: r = 0.74 [P < .001]; right DN: r = 0.48 [P = .03]). Conclusion The proposed model accurately and efficiently segmented the DN from brain QSM images. The model is publicly available (https://github.com/art2mri/DentateSeg). Keywords: MR Imaging, Brain/Brain Stem, Segmentation, Convolutional Neural Network, Supervised Learning, Computer Applications-3D, Volume Analysis, Image Postprocessing ClinicalTrials.gov registration no. NCT04349514 Supplemental material is available for this article. © RSNA, 2025.