Normalization Techniques in Training DNNs: Methodology, Analysis and Application.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Normalization techniques are crucial for accelerating deep neural network (DNN) training and enhancing model generalization.
• These methods have demonstrated success across a wide range of applications.

Approach:

• This paper provides a comprehensive review of normalization methods in DNN training, examining their past, present, and future.
• A unified perspective on the optimization motivations behind various approaches is presented.
• A taxonomy is introduced to clarify similarities and differences among normalization techniques.

Key Points:

• Representative normalizing activation methods are decomposed into three core components: normalization area partitioning, normalization operation, and normalization representation recovery.
• This decomposition offers valuable insights for the development of novel normalization techniques.
• The review discusses the current understanding of normalization methods and their specific applications.

Conclusions:

• Normalization is vital for efficient and effective DNN training.
• Understanding the components of normalization aids in designing improved methods.
• Normalization effectively addresses key challenges in various DNN applications.