Denoising cosine similarity: A theory-driven approach for efficient representation learning.

Area of Science:

• Machine Learning
• Data Science
• Computer Vision
• Speech Processing

Background:

• Representation learning is crucial for efficient machine learning across various tasks.
• Real-world datasets often contain noise, degrading the quality of learned representations.
• Existing methods inadequately address noise contamination during representation learning.

Purpose of the Study:

• To develop a method for learning robust representations resistant to dataset noise.
• To propose a novel loss function that integrates denoising capabilities into representation learning.

Main Methods:

• Introduction of the denoising Cosine-Similarity (dCS) loss, a modification of cosine-similarity loss.
• Theoretical and empirical validation of the dCS loss's denoising properties.
• Development of implementable estimators for the dCS loss with statistical guarantees.

Main Results:

• The proposed dCS loss demonstrates superior performance compared to baseline objective functions.
• Empirical evidence confirms the effectiveness of dCS loss in both vision and speech domains.
• Learned representations exhibit enhanced robustness against noise in the training data.

Conclusions:

• The dCS loss offers an effective solution for learning robust representations from noisy datasets.
• This approach significantly improves representation quality and downstream task performance in noisy conditions.
• The dCS loss provides a valuable tool for practical machine learning applications dealing with imperfect data.