Bayesian Weight Decay on Bounded Approximation for Deep Convolutional Neural Networks

Area of Science:

• Machine Learning
• Deep Learning
• Computer Vision

Background:

• Determining the optimal weight decay parameter for deep convolutional neural networks (CNNs) is crucial for achieving good generalization.
• Traditional numerical trials for finding this parameter are computationally expensive, especially for large CNN architectures.

Purpose of the Study:

• To develop an efficient analytical solution for calculating the weight decay parameter in deep CNNs.
• To reduce the computational cost associated with hyperparameter tuning for improved model generalization.

Main Methods:

• Formulation of an analytical solution using a proposed objective function and Bayesian probability distributions.
• Development of a novel approximation method utilizing limited Hessian matrix information.
• Implementation of a linear time complexity algorithm for calculating the approximate solution.

Main Results:

• The approximate solution is theoretically guaranteed by a provable bound.
• The proposed method achieves linear time complexity concerning CNN depth and width.
• Experimental validation on real-world image datasets demonstrates suitability and significant time cost reduction.
• Achieved good classification performance with reduced training time.

Conclusions:

• The proposed analytical and approximation methods efficiently determine the optimal weight decay parameter for deep CNNs.
• This approach significantly lowers the computational burden of hyperparameter tuning, enabling faster investigation of deep learning models.
• The method is effective for real-world image datasets, leading to improved generalization and classification performance.