MLR-SNet: Transferable LR Schedules for Heterogeneous Tasks.

Area of Science:

• Deep Learning
• Machine Learning Optimization
• Hyperparameter Tuning

Background:

• Learning rate (LR) schedules are crucial for training deep neural networks (DNNs) using stochastic gradient descent (SGD).
• Current hand-designed LR schedules lack adaptability to diverse non-convex optimization problems and require task-specific retraining.
• Significant effort is needed to find optimal LR schedules for new tasks with varying data modalities, architectures, or capacities.

Purpose of the Study:

• To propose a novel, flexible, and transferable method for automatically generating effective learning rate schedules for DNNs.
• To address the limitations of fixed, hand-designed LR schedules in adapting to dynamic training processes.
• To enable plug-and-play generalization of learned LR schedules across heterogeneous tasks.

Main Methods:

• Parameterizing LR schedules using an explicit mapping formulation named MLR-SNet.
• Utilizing a learnable parameterized structure for MLR-SNet to adapt to DNN training dynamics.
• Conducting benchmark experiments on image and text classification tasks.

Main Results:

• MLR-SNet successfully generates effective LR schedules that comply with DNN training dynamics.
• Meta-learned LR schedules demonstrate transferability and plug-and-play capability across diverse tasks (e.g., different epochs, architectures, datasets).
• The method shows robustness against noisy training data and achieves convergence rates comparable to state-of-the-art methods.

Conclusions:

• MLR-SNet offers a flexible and adaptive solution for learning rate scheduling in deep learning.
• The proposed method generalizes well to new, heterogeneous tasks, outperforming or matching hand-designed schedules.
• MLR-SNet provides a robust and theoretically sound approach to optimizing DNN training.