Data coarse graining can improve model performance

Area of Science:

• Machine Learning
• Statistical Physics
• Data Science

Background:

• Lossy data transformations typically discard information.
• Techniques like data pruning and lossy data augmentation can enhance machine learning generalization.
• Understanding the underlying mechanisms of this paradox is crucial for developing effective ML strategies.

Purpose of the Study:

• To investigate the paradoxical phenomenon of lossy data transformations improving generalization in machine learning.
• To analyze the impact of data coarse-graining on prediction risk using a solvable model.
• To provide an analytical explanation for the effectiveness of data augmentation in machine learning.

Main Methods:

• Utilized a solvable model of high-dimensional, ridge-regularized linear regression.
• Employed coarse-graining schemes inspired by renormalization group methods in statistical physics.
• Analyzed feature relevance and systematic discarding of features based on their importance to the learning task.

Main Results:

• Discovered a nonmonotonic relationship between the degree of data coarse-graining and prediction risk.
• Demonstrated that a high-pass filtering scheme, removing low-signal features, improves model generalization.
• Showed that a low-pass scheme, retaining high-signal features, is detrimental to performance.

Conclusions:

• Careful data augmentation, by removing less relevant information, can isolate predictive signals and improve generalization.
• The observed nonmonotonicity in prediction risk is a genuine effect of data coarse-graining, not an artifact of other phenomena.
• Statistical physics principles offer a valuable framework for understanding complex machine learning behaviors.