Instance-dependent Early Stopping for Adaptive Data Pruning

Area of Science:

• Machine Learning
• Deep Learning Optimization
• Computational Efficiency

Background:

• Conventional early stopping halts model training based on overall validation performance, potentially leading to overfitting and wasted computation on already learned instances.
• Existing methods lack instance-level granularity, failing to account for individual data point mastery during the training process.

Purpose of the Study:

• To introduce Instance-dependent Early Stopping (IES) for more efficient model training by adapting early stopping to the instance level.
• To accelerate training and reduce computational overhead by identifying and excluding mastered instances from backpropagation.

Main Methods:

• IES identifies mastered instances by analyzing the second-order differences of their loss values, using a uniform criterion applicable across all samples.
• An enhanced variant, IES+, further optimizes the forward pass for aggressive training acceleration.
• The method was evaluated on supervised fine-tuning of large language models.

Main Results:

• IES reduces the number of instances undergoing backpropagation by 10%-50%, leading to faster training while maintaining or improving model performance.
• IES+ achieves significant reductions in wall-clock time, prioritizing speed.
• IES demonstrates effectiveness in supervised fine-tuning of large language models, yielding substantial computational savings and performance preservation.

Conclusions:

• Instance-dependent Early Stopping (IES) offers a novel and effective approach to accelerate deep learning training and enhance computational efficiency.
• IES provides a scalable solution applicable to various machine learning tasks, including the fine-tuning of large language models.
• The proposed method balances computational savings with model performance, making it a valuable tool for resource-constrained environments.