Revisiting Internal Covariate Shift for Batch Normalization.

Area of Science:

• Deep Learning
• Machine Learning Optimization

Background:

• Batch Normalization (BatchNorm) is widely used in deep learning.
• Its success is often attributed to reducing internal covariate shift (ICS).
• Recent studies question the ICS hypothesis, proposing alternative explanations like enabling higher learning rates and smoothing optimization landscapes.

Purpose of the Study:

• To empirically validate the importance of ICS reduction for BatchNorm's success.
• To investigate alternative explanations for BatchNorm's effectiveness.
• To analyze the role of BatchNorm parameters in performance and ICS.

Main Methods:

• Empirical validation of ICS reduction's importance.
• Demonstration of alternative properties without performance gains.
• Analysis of BatchNorm parameters and their connection to ICS.
• Comparison with a novel normalization scheme lacking ICS reduction.

Main Results:

• Alternative properties (high learning rates, smoothed landscapes) do not guarantee performance gains without ICS reduction.
• A normalization scheme with alternative properties but no ICS reduction showed poor performance.
• All tested BatchNorm variants were found to reduce ICS.
• BatchNorm parameters' effectiveness was visualized and linked to ICS.

Conclusions:

• The primary reason for BatchNorm's success is its ability to reduce internal covariate shift (ICS).
• Alternative explanations for BatchNorm's effectiveness are insufficient on their own.
• ICS reduction remains a critical factor for achieving high performance in deep learning models.