Boundary between noise and information applied to filtering neural network weight matrices

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Deep neural networks (DNNs) often exhibit overparameterization, leading to weight matrices with partially random characteristics.
• The singular value spectrum of these matrices shows a boundary between random components and learned information, comparable to the Porter-Thomas distribution.

Purpose of the Study:

• To introduce a novel noise filtering algorithm for deep neural networks.
• To mitigate the impact of random noise within the singular value spectrum of DNN weight matrices.
• To enhance the generalization performance of DNNs, particularly when trained with noisy labels.

Main Methods:

• Analysis of singular value spectrum in overparameterized deep neural networks.
• Comparison of singular vectors to the Porter-Thomas distribution to identify a randomness-information boundary.
• Development of a noise filtering algorithm that removes small singular values and reduces large ones.

Main Results:

• The proposed noise filtering algorithm effectively counteracts level repulsion between noise and information in the singular value spectrum.
• Significant improvements in generalization performance were observed for networks trained with label noise after applying the filtering algorithm.

Conclusions:

• Noise filtering is a viable strategy to enhance DNN generalization by addressing the spectral properties of weight matrices.
• The algorithm's effectiveness is particularly pronounced in scenarios involving noisy training data, highlighting its practical utility.