Exploring the Stochastic Regularisation in Normalisation Layers for Semi-Supervised Learning

Area of Science:

• Deep Learning
• Machine Learning
• Computer Vision
• Natural Language Processing
• Audio Processing

Background:

• Semi-supervised learning (SSL) reduces reliance on labeled data but often overlooks architectural suitability.
• Existing normalization layers like Batch Normalization (BN), Group Normalization (GN), and Layer Normalization (LN) have limitations in SSL.
• Unrestricted stochastic regularization from BN can degrade performance with mismatched label distributions; deterministic methods like GN/LN can be suboptimal.

Purpose of the Study:

• To investigate the impact of normalization layer stochastic regularization on semi-supervised learning.
• To propose novel normalization techniques that introduce controllable randomness for improved SSL performance.
• To address confirmation bias and enhance the robustness and effectiveness of SSL models.

Main Methods:

• Analysis of stochastic regularization's impact on optimization gradient stability in SSL.
• Proposal of Shuffle Layer Normalisation (SLN) and Shuffle Group Normalisation (SGN) introducing controllable randomness.
• Experimental validation across diverse datasets (image, text, audio) using state-of-the-art SSL algorithms.

Main Results:

• The degree of stochastic regularization critically affects optimization gradient stability and SSL performance.
• SLN/SGN introduce controllable randomness into deterministic normalization layers without increasing parameters.
• Experiments show significant performance enhancement of SSL algorithms using SLN/SGN across modalities.

Conclusions:

• A moderate level of stochastic regularization is crucial for robust convergence and generalization in SSL.
• SLN/SGN offer a novel approach to enhance SSL by providing tunable regularization.
• The proposed methods are compatible with pre-trained parameters and various backbone architectures (CNNs, Transformers).