Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning

Area of Science:

• Deep Learning
• Machine Learning Optimization
• Artificial Intelligence

Background:

• Existing neural network optimizers primarily focus on loss reduction.
• They do not adequately address the distinct requirements of training from scratch versus fine-tuning pre-trained models.
• Big data and pre-trained models necessitate specialized optimization strategies.

Purpose of the Study:

• To propose DualOpt, a novel optimization approach decoupling techniques for distinct neural network training paradigms.
• To enhance convergence and generalization for training from scratch.
• To improve fine-tuning performance by mitigating knowledge forgetting in pre-trained models.

Main Methods:

• Introduced real-time layer-wise weight decay for training from scratch.
• Integrated weight rollback into the optimizer for fine-tuning.
• Extended layer-wise weight decay to dynamically adjust rollback levels across layers.

Main Results:

• DualOpt demonstrated state-of-the-art performance across diverse tasks.
• Experiments included image classification, object detection, semantic segmentation, and instance segmentation.
• The approach showed broad applicability and effectiveness.

Conclusions:

• DualOpt offers a specialized optimization framework for different neural network training scenarios.
• The method successfully enhances both training from scratch and fine-tuning.
• This work advances neural network optimization techniques for big data and pre-trained models.