Dataset Pruning: Reducing Training Data by Examining SGD-Influence

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Deep learning's success depends on large datasets, leading to significant computational and infrastructure costs.
• Key questions arise regarding data contribution to model performance and generalization.
• The need for efficient methods to create smaller, representative training sets is critical.

Purpose of the Study:

• To develop an optimization-based method for dataset pruning, selecting the most influential training samples.
• To construct the smallest possible training data subset while maintaining a controlled generalization gap.
• To efficiently estimate the influence of individual training samples on model generalization.

Main Methods:

• Dataset pruning: An optimization-based sample selection technique.
• SGD-Influence method: Tracks parameter changes during stochastic gradient descent to estimate sample influence, bypassing traditional limitations.
• Distributed discrete optimization: Partitions datasets into manageable segments for efficient processing.

Main Results:

• Dataset pruning effectively identifies influential training samples and constructs minimal proxy datasets.
• Empirical generalization gaps align with theoretical predictions.
• The proposed method outperforms state-of-the-art approaches in efficiency and accuracy.
• Achieved a 61.26% reduction in computational cost compared to previous work with improved accuracy.

Conclusions:

• Dataset pruning offers a computationally efficient and theoretically grounded approach to data selection in deep learning.
• The SGD-Influence and distributed optimization methods enable effective sample influence estimation and efficient subset construction.
• This work provides a pathway to significantly reduce the costs associated with deep learning training while enhancing model performance.