Diminishing Batch Normalization.

Area of Science:

• Machine Learning
• Deep Learning
• Artificial Intelligence

Background:

• Batch Normalization (BN) is a widely adopted technique for accelerating neural network training.
• Existing methods like BN have limitations in certain training scenarios.
• The need for improved normalization techniques in deep learning persists.

Purpose of the Study:

• To propose and analyze a generalized Batch Normalization algorithm called Diminishing Batch Normalization (DBN).
• To investigate the convergence properties of the DBN algorithm.
• To demonstrate the practical effectiveness of DBN compared to existing normalization methods.

Main Methods:

• Introduced a weighted averaging update for trainable parameters within the BN framework, creating DBN.
• Provided a theoretical convergence analysis for DBN, applicable to models with arbitrary activation functions.
• Conducted numerical experiments using complex Convolutional Neural Networks (CNNs) and Feedforward Neural Networks (FNNs) on diverse datasets.

Main Results:

• DBN converges to a stationary point with respect to trainable parameters.
• Theoretical analysis of DBN's convergence is presented, generalizable to standard BN.
• Empirical results show DBN outperforms original BN and Layer Normalization (LN) on multiple datasets and model architectures.

Conclusions:

• DBN offers a robust generalization of BN with improved performance.
• The theoretical convergence analysis provides novel insights into BN-based algorithms.
• DBN demonstrates superior efficacy in accelerating deep neural network training across various tasks.