Gradient-guided layerwise adaptive noise injection for pre-trained language model fine-tuning.

Area of Science:

• Natural Language Processing
• Machine Learning
• Deep Learning

Background:

• Fine-tuning pre-trained language models (PLMs) enhances performance on downstream tasks.
• Overfitting is a significant challenge during PLM fine-tuning, especially in low-resource scenarios.
• Current noise-based regularization methods lack adaptability to dynamic training states.

Purpose of the Study:

• To propose a novel fine-tuning framework, gradient-guided noise injection (GNI).
• To address the limitations of static regularization methods in PLM fine-tuning.
• To improve model performance and robustness in low-resource settings.

Main Methods:

• Introduced gradient-guided noise injection (GNI) for dynamic regularization.
• Utilized gradient information for real-time, layer-wise noise intensity adjustment.
• Conducted extensive experiments on GLUE and SuperGLUE benchmarks.

Main Results:

• GNI demonstrated improved average performance across various pre-trained models and tasks.
• The proposed method exhibited stable generalizability and robustness.
• GNI effectively mitigates overfitting in low-resource fine-tuning scenarios.

Conclusions:

• GNI offers a simple yet effective solution for optimization-state aware dynamic regularization.
• The framework enhances the fine-tuning process for pre-trained language models.
• Gradient-guided noise injection represents a significant advancement in regularization techniques.