Variational HyperAdam: A Meta-Learning Approach to Network Training.

Area of Science:

• Deep learning
• Machine learning optimization

Background:

• Stochastic optimization is widely used for training deep neural networks.
• Learning-based optimizers show promise but often lack generalization due to their black-box nature and reliance on meta-training.

Purpose of the Study:

• To propose Variational HyperAdam, a novel optimizer that combines the strengths of generalized Adam (HyperAdam) within a variational framework.
• To improve the generalization ability of learning-based optimizers by leveraging human-designed optimizer principles.

Main Methods:

• Variational HyperAdam treats neural network parameter updates as random variables, predicting their approximate posterior distribution.
• It employs a learnable generalized Adam for expectation estimation and a VarBlock for variance estimation.
• The optimizer is learned using a meta-learning approach with a meta-training loss derived from variational inference.

Main Results:

• Variational HyperAdam achieved state-of-the-art performance in training various neural network architectures.
• Effective across diverse datasets and network types, including multilayer perceptrons, CNNs, LSTMs, and ResNets.
• Demonstrated superior network training efficiency and generalization compared to existing methods.

Conclusions:

• Variational HyperAdam offers a significant advancement in neural network optimization.
• The proposed variational framework effectively integrates learning-based and human-designed optimization strategies.
• This approach enhances the performance and generalizability of deep learning model training.