Training Faster by Separating Modes of Variation in Batch-Normalized Models.

Area of Science:

• Deep Learning
• Machine Learning
• Computer Vision

Background:

• Batch Normalization (BN) is crucial for training deep Convolutional Neural Networks (CNNs), stabilizing training by normalizing layer outputs with mini-batch statistics.
• BN accelerates training by enabling larger learning rates and reducing sensitivity to parameter initialization.
• Existing BN methods assume a single probability distribution for mini-batch samples, which may not hold true due to non-linearities in CNNs.

Purpose of the Study:

• To investigate Batch Normalization (BN) through the lens of Fisher kernels derived from generative probability models.
• To propose an improved normalization technique, Mixture Normalization (MN), that better handles asymmetric data distributions in deep networks.
• To evaluate the effectiveness of MN in accelerating training and improving accuracy for both discriminative (CNNs) and generative (GANs) models.

Main Methods:

• Analyzed BN as equivalent to the Fisher vector of a Gaussian distribution under specific assumptions.
• Proposed Mixture Normalization (MN) by approximating data distributions with Gaussian Mixture Models (GMMs) to handle asymmetric layer outputs.
• Implemented and experimentally validated MN by replacing BN layers in CNN architectures (Inception-V3, DenseNet) and Generative Adversarial Networks (DCGAN).

Main Results:

• MN reduced the number of gradient updates needed to reach maximum test accuracy by 31%-47% compared to BN on CIFAR-10/100 datasets.
• Replacing even a few BN modules with MN in deep CNNs (Inception-V3, DenseNet) led to significant training acceleration and improved final test accuracy.
• In GANs, MN accelerated training by ~58% and achieved a lower Fréchet Inception Distance (FID) score (33.35 vs. 37.56) compared to standard BN.

Conclusions:

• Mixture Normalization (MN) offers a principled way to enhance Batch Normalization (BN) by accounting for complex data distributions.
• MN provides substantial benefits in training speed and model performance across various deep learning architectures, including CNNs and GANs.
• The proposed GMM-based normalization approach effectively addresses limitations of standard BN, particularly in scenarios with asymmetric output distributions.