Effective and Efficient Batch Normalization Using a Few Uncorrelated Data for Statistics Estimation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Deep neural networks (DNNs) are crucial in modern AI.
• Batch Normalization (BN) is vital for DNN training but computationally expensive.
• Current BN methods hinder training speed due to parallelization challenges.

Purpose of the Study:

• To propose a cost-effective methodology for batch normalization in DNNs.
• To enhance training speed without compromising model accuracy or convergence.
• To address the computational burden of BN in deep learning.

Main Methods:

• Developed techniques using minimal sampled or generated data for mean and variance estimation.
• Introduced batch sampling (BS) and feature sampling (FS) for uncorrelated data.
• Proposed virtual data set normalization (VDN) for synthetic data generation.
• Designed multiway strategies to balance normalization effectiveness and execution efficiency.

Main Results:

• Achieved practical 1.98x BN layer acceleration and 23.2% overall training speedup on GPUs.
• Demonstrated negligible loss in model accuracy and convergence rate.
• Showcased effective solutions for the 'micro-BN' problem with tiny batch sizes.
• Validated methods across various DNN models without specialized libraries.

Conclusions:

• The proposed methods offer a promising solution for efficient DNN training.
• Significant speedups are achievable with minimal impact on performance.
• These techniques are effective even in challenging scenarios like small batch sizes.