Exploring the Stochastic Regularisation in Normalisation Layers for Semi-Supervised Learning

Semi-supervised learning is essential for deep learning models as it can effectively reduce reliance on large amounts of labelled data. However, research in this field often prioritises training strategies, overlooking whether existing model architectures are inherently suitable for semi-supervised tasks. We noticed this oversight and begin our analysis with the normalisation layers. Specifically, we found that the stochastic regularisation introduced by Batch Normalisation (BN) layers does not always yields better performance. In fact, when labelled and unlabelled data have mismatched label distributions, such unrestricted stochastic regularisation can even degrade the model's performance. Conversely, layers like Group Normalisation (GN) and Layer Normalisation (LN), which eliminate stochastic effects, may also perform suboptimally. We show that the degree of stochastic regularisation impacts the stability of the optimisation gradient, highlighting the need for a *moderate* level of stochastic regularisation in semi-supervised learning to achieve robust convergence and generalisation. Otherwise, techniques like pseudo-labelling in semi-supervised learning are prone to generating incorrect pseudo-labels, exacerbating the issue of confirmation bias. To address this problem, we propose novel normalisation techniques termed Shuffle Layer Normalisation and Shuffle Group Normalisation (SLN/SGN) that introduce controllable randomness into LN/GN without increasing model parameters, thereby making semi-supervised learning more robust and effective. Through experiments across diverse datasets, including image, text, and audio modalities, we demonstrate that SLN/SGN significantly enhances the performance of state-of-the-art semi-supervised learning algorithms. Notably, SLN/SGN can reuse pre-trained parameters and is compatible with most existing backbone models, including CNNs and Transformers.