Learning from Data with Heterogeneous Noise using SGD.

Area of Science:

• Machine Learning
• Data Science
• Optimization Algorithms

Background:

• Learning from data of variable quality from heterogeneous sources presents significant challenges.
• Existing methods struggle with the full generality of heterogeneous data.
• A common scenario involves data observed through heterogeneous noise, where noise level indicates data quality.

Purpose of the Study:

• To investigate the impact of heterogeneous noise on the performance of stochastic gradient algorithms.
• To develop and analyze methods for learning from data with varying noise levels.
• To address specific applications like learning with local differential privacy and learning from variable-quality labels.

Main Methods:

• Utilizing stochastic gradient descent (SGD) algorithms within a model of heterogeneous noise.
• Analyzing the influence of noise levels on data source order in SGD.
• Proposing a novel method for dynamically adjusting the learning rate based on noise heterogeneity.
• Deriving new regret bounds for the proposed method in specific scenarios.

Main Results:

• The order of using datasets with heterogeneous noise in standard SGD is sensitive to the learning rate.
• A method for adapting the learning rate to noise heterogeneity was proposed and analyzed.
• New regret bounds were proven for the proposed adaptive learning rate method.
• Empirical results demonstrated superior performance compared to fixed learning rates or using only the least noisy dataset, especially at low to moderate noise levels.

Conclusions:

• Heterogeneous noise significantly impacts the performance of learning algorithms.
• Adaptive learning rate strategies are crucial for effectively learning from heterogeneous data.
• The proposed method offers a practical and effective solution for improving machine learning performance in the presence of variable data quality.