L1 -Norm Batch Normalization for Efficient Training of Deep Neural Networks.

Area of Science:

• Deep Learning
• Computer Engineering
• Artificial Intelligence

Background:

• Batch Normalization (BN) is crucial for deep neural network (DNN) training but introduces computational overhead and memory usage.
• The nonlinear operations in conventional L2-norm BN (L2BN) hinder low bit-width quantization, limiting hardware efficiency.
• Existing BN methods present challenges for resource-constrained deep learning hardware and mobile applications.

Purpose of the Study:

• To introduce a novel L1-norm Batch Normalization (L1BN) that utilizes only linear operations.
• To demonstrate L1BN's equivalence to L2BN with a scaling factor for comparable performance.
• To enhance computational efficiency and hardware friendliness in DNN training.

Main Methods:

• Developed L1BN employing solely linear operations in forward and backward propagation.
• Validated L1BN's performance against L2BN on various convolutional neural networks and generative adversarial networks.
• Assessed L1BN's efficiency in application-specific integrated circuit synthesis with resource constraints.

Main Results:

• L1BN achieves comparable performance and convergence rates to L2BN.
• L1BN offers significant computational efficiency improvements over L2BN.
• In hardware synthesis, L1BN demonstrated a 25% speedup and 37% energy saving compared to L2BN.

Conclusions:

• L1BN provides a hardware-friendly alternative to L2BN, enhancing computational efficiency and simplifying accelerator design.
• The proposed method facilitates fully quantized DNN training, enabling advanced AI capabilities on mobile devices.
• L1BN supports transfer and continual learning for future AI applications in resource-limited environments.