Student Loss: Towards the Probability Assumption in Inaccurate Supervision

Area of Science:

• Machine Learning
• Computer Science
• Artificial Intelligence

Background:

• Learning with noisy labels presents a significant challenge in dataset curation.
• Existing methods often treat mislabeled and clean samples indiscriminately, limiting robustness.
• Natural discrepancies between clean and mislabeled data are often overlooked.

Purpose of the Study:

• To develop a novel method for improving learning robustness in the presence of noisy labels.
• To leverage the properties of the student distribution for data selection and noise resistance.
• To introduce a metric learning strategy for enhanced performance in inaccurate supervision scenarios.

Main Methods:

• Proposed a new loss function, termed student loss, based on the assumption that deep features with the same label follow a student distribution.
• Embedded the student distribution into the learning process to exploit the sharpness of its curve for data selection.
• Developed a large-margin student (LT) loss by incorporating a metric learning strategy.
• Introduced a novel approach using prior probability assumptions in feature representation to reduce the impact of mislabeled samples.

Main Results:

• The student loss method demonstrates natural data-selectivity, causing clean samples to aggregate tightly and mislabeled samples to scatter.
• The proposed LT loss significantly enhances the capability to resist mislabeled samples.
• The approach effectively decreases the contributions of mislabeled samples, even outperforming existing robust losses.
• Experiments show substantial performance improvements, exceeding 50% in some conditions, particularly under inaccurate supervision.

Conclusions:

• The student loss framework offers an effective strategy for learning with noisy labels by modeling feature distributions.
• The LT loss provides a powerful enhancement for noisy label learning, outperforming state-of-the-art methods.
• This work pioneers the use of prior probability assumptions in feature representation for noise reduction in machine learning.