Data coarse graining can improve model performance

Area of Science:

• Machine Learning
• Statistical Physics
• Data Science

Background:

• Lossy data transformations typically discard information.
• Techniques like data pruning and lossy data augmentation can paradoxically improve machine learning generalization.
• Understanding the mechanisms behind this phenomenon is crucial for developing more effective ML models.

Purpose of the Study:

• To investigate the paradox of information loss improving generalization in machine learning.
• To analyze the impact of data coarse-graining on prediction risk using a solvable model.
• To provide an analytical explanation for the benefits of certain data augmentation strategies.

Main Methods:

• Studied high-dimensional, ridge-regularized linear regression under data coarse-graining.
• Employed schemes inspired by the renormalization group to systematically discard features based on relevance.
• Analyzed the dependence of prediction risk on the degree of coarse-graining.

Main Results:

• Discovered a nonmonotonic relationship between the degree of data coarse-graining and prediction risk.
• A high-pass coarse-graining scheme, filtering out low-signal features, improved generalization.
• A low-pass scheme, integrating out high-signal features, proved detrimental.
• Demonstrated that this nonmonotonicity is a distinct effect of data coarse-graining, not an artifact of double descent.

Conclusions:

• Careful data augmentation, by stripping irrelevant degrees of freedom, can isolate more predictive signals and enhance model generalization.
• The study highlights a complex, nonmonotonic risk landscape influenced by data structure.
• Statistical physics principles offer a valuable framework for understanding modern machine learning phenomena.