Compressing Features for Learning With Noisy Labels.

Area of Science:

• Machine Learning
• Computer Vision
• Deep Learning

Background:

• Supervised learning struggles with noisy labels, leading to model overfitting and poor generalization.
• Existing methods often fail to effectively handle corrupted data, hindering performance on clean datasets.

Purpose of the Study:

• To introduce compression inductive bias into network architectures to mitigate overfitting caused by noisy labels.
• To enhance the robustness and generalization capabilities of models trained on datasets with label noise.

Main Methods:

• Revisiting and applying Dropout and Nested Dropout as compression constraints to network architectures.
• Integrating compression regularization with co-teaching strategies for improved performance.
• Conducting theoretical bias-variance decomposition to analyze the impact of compression regularization.

Main Results:

• Compression regularization, particularly Nested Dropout, effectively reduces overfitting in the presence of noisy labels.
• The proposed approach, combined with co-teaching, achieves state-of-the-art or comparable performance on real-world noisy datasets (Clothing1M, ANIMAL-10N).
• Theoretical analysis confirms that feature compression combats label noise via an information bottleneck formulation.

Conclusions:

• Compression inductive bias is a viable strategy for learning with noisy labels.
• The integration of compression regularization with co-teaching offers a significant performance boost.
• The developed methods provide a simple yet effective solution for improving model generalization on noisy datasets.