Re-Thinking the Effectiveness of Batch Normalization and Beyond | JoVE Visualize

Area of Science:

Deep Learning
Neural Network Optimization

Background:

Batch Normalization (BN) accelerates deep neural network training but its effectiveness mechanism is debated, with recent focus on Lipschitzness.
Existing research questions if Lipschitzness fully explains BN's benefits and if vanilla BN can be further optimized.

Purpose of the Study:

To investigate the underlying mechanisms of Batch Normalization (BN) effectiveness in deep neural networks.
To propose and validate a novel normalization technique, Complete Batch Normalization (CBN), for improved training dynamics and performance.

Main Methods:

Theoretical analysis of Stochastic Gradient Descent (SGD) on non-convex problems, identifying three key convergence-enhancing effects.
Development of Complete Batch Normalization (CBN) by modifying BN's structure and normalization placement.
Empirical validation using extensive experiments on CIFAR10, CIFAR100, and ILSVRC2012 datasets with diverse activation functions.

Main Results:

Vanilla BN with ReLU induces three beneficial effects for convergence (gradient Lipschitz constant reduction, reduced squared stochastic gradient expectation, and reduced stochastic gradient variance), but not with other activations.
Complete Batch Normalization (CBN) consistently elicits all three convergence-enhancing effects, irrespective of the activation function used.
CBN demonstrates faster training convergence, leads to smaller local minima, and significantly boosts test accuracy across multiple activation functions (Sigmoid, Tanh, ReLU, SELU, Swish), with notable improvements for Sigmoid, Tanh, and SELU.

Conclusions:

The effectiveness of BN stems from inducing specific gradient properties, not solely Lipschitzness.
Complete Batch Normalization (CBN) offers a theoretically grounded and empirically validated improvement over vanilla BN, enhancing training stability and performance across diverse network architectures and activations.
CBN enables high performance with traditionally less effective activation functions, broadening the applicability of deep learning models.