Random effects during training: Implications for deep learning-based medical image segmentation.

Area of Science:

• Medical image analysis
• Deep learning
• Computational neuroscience

Background:

• Deep learning models for image segmentation exhibit performance variability due to random training effects.
• This variability can impact the reliability of standard model comparison methods.

Purpose of the Study:

• To assess the impact of random training effects on the reliability of comparing deep learning segmentation models.
• To evaluate the effectiveness of common statistical methods in detecting true performance differences.

Main Methods:

• Utilized nnU-Net with 50 random seeds across three 3D medical image segmentation tasks (brain tumor, hippocampus, cardiac).
• Assessed segmentation performance using hold-out validation and 5-fold cross-validation.
• Employed Paired t-test and Wilcoxon signed rank test on Dice scores to measure statistical significance.

Main Results:

• The top-performing seed significantly outperformed 0-76% of other seeds with hold-out validation.
• With 5-fold cross-validation, the top seed outperformed 10-38% of other seeds.
• High rates of statistically significant differences were observed between models trained with the same algorithm.

Conclusions:

• Random training effects can lead to frequent statistically significant performance differences.
• Statistical significance is a weak and unreliable indicator of genuine performance superiority between models from the same deep learning algorithm.