Switchable Normalization for Learning-to-Normalize Deep Representation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Normalization is crucial for deep neural network performance.
• Existing normalization techniques have limitations with varying batch sizes and architectures.
• Selecting the appropriate normalization scope (channel, layer, minibatch) is challenging.

Purpose of the Study:

• To introduce Switchable Normalization (SN), a novel method for learning normalization strategies in deep neural networks.
• To enable each normalization layer to adaptively select its optimal normalization scope.
• To improve the robustness and performance of deep learning models across diverse tasks and datasets.

Main Methods:

• Proposing Switchable Normalization (SN) that learns importance weights for channel, layer, and minibatch scopes.
• Implementing an end-to-end learning approach for adaptive normalization.
• Evaluating SN across various network architectures and challenging benchmarks.

Main Results:

• SN demonstrates adaptability to diverse network architectures and tasks.
• The method maintains high performance even with small minibatch sizes.
• SN outperforms existing normalization techniques on benchmarks like ImageNet, COCO, and Kinetics.

Conclusions:

• Switchable Normalization offers a flexible and robust solution for deep learning normalization.
• The adaptive nature of SN simplifies hyper-parameter tuning compared to methods like group normalization.
• SN provides insights into the interplay between normalization choices, tasks, and datasets.