A systematic evaluation of learning rate policies in training CNNs for brain tumor segmentation

Convolutional neural networks (CNNs) have recently emerged as a powerful approach for automatic segmentation of brain tumor subregions on 3D multi-parametric MRI scans. Learning rate is a crucial hyperparameter in the training of CNNs, impacting the performance of the learned model. Different learning rate policies trace unique trajectories in the optimization landscape that converge to local minima with varying generalization properties. In this work, we empirically evaluated nine learning rate policy-optimizer pairs with two state-of-the-art architectures, namely 2D slice-based U-Net and 3D DeepMedicRes, on an augmented brain tumor dataset of 534 subjects. Segmentation performance was quantified in terms of Dice similarity coefficient and Hausdorff distance metrics. The policies were ranked based on the final ranking score (FRS) employed by the BraTS challenge, with the statistical significance of the rankings evaluated by random permutation test. For 2D slice-based U-Net architecture, an overall ranking of learning rate policies showed that the polynomial decay policy with Adam optimizer significantly outperformed other policies for the task of individual and hierarchical segmentation of tumor subregions (p< 10^-4). For 3D segment-based DeepMedicRes architecture, polynomial decay policy with Adam optimizer performed significantly better than all other policies, with the exception of polynomial decay with SGD optimizer for the same task (p< 10^-4). Based on the FRS, polynomial decay policy with Adam and SGD optimizer occupied the top two positions respectively, but the difference was not statistically significant (p> 0.3). These findings were also validated on the BraTS 2019 Validation dataset which comprised of an additional 125 subjects.