Revisiting Internal Covariate Shift for Batch Normalization

Despite the success of batch normalization (BatchNorm) and a plethora of its variants, the exact reasons for its success are still shady. The original BatchNorm article explained it as a mechanism that reduces the internal covariate shift (ICS), i.e., the distribution shifts in the input of the layers during training. Recently, some articles manifested skepticism on this hypothesis and provided alternative explanations for the success of BatchNorm, such as the applicability of very high learning rates and the ability to smooth the landscape in optimization. In this work, we counter these alternative arguments by demonstrating the importance of reduction in ICS following an empirical approach. We demonstrated various ways to achieve the abovementioned alternative properties without any performance boost. In this light, we explored the importance of different BatchNorm parameters (i.e., batch statistics and affine transformation parameters) by visualizing their effectiveness in the performance and analyzed their connections with ICS. Afterward, we showed a different normalization scheme that fulfills all the alternative explanations except reduction in ICS. Despite having all the alternative properties, we observed its poor performance, which nullifies the alternative claims, rather signifies the importance of the ICS reduction. We performed comprehensive experiments on many variants of BatchNorm, finding that all of them similarly reduce ICS.