Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networks.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Statistics

Background:

• Training stochastic neural networks presents challenges, especially within the PAC-Bayesian framework.
• Existing methods may involve surrogate losses or looser bounds, impacting direct optimization and generalization guarantees.

Purpose of the Study:

• To develop novel methods for training stochastic neural networks, particularly in a PAC-Bayesian setting.
• To introduce partially-aggregated estimators through ensemble averaging and derive a directly optimizable PAC-Bayesian objective.

Main Methods:

• Ensemble averaging of stochastic neural networks to create partially-aggregated estimators.
• Reformulation of a PAC-Bayesian bound for signed-output networks.
• Development of a differentiable objective function for direct optimization.
• Empirical evaluation of the proposed methods against existing approaches.

Main Results:

• Partially-aggregated estimators provide unbiased, lower-variance output and gradient estimates.
• The reformulated PAC-Bayesian bound yields a directly optimizable, differentiable objective.
• The proposed method achieves competitive generalization guarantees.
• A simpler PAC-Bayesian training scheme for sign-activation networks is demonstrated.

Conclusions:

• Ensemble averaging offers a powerful technique for enhancing the training of stochastic neural networks.
• The derived PAC-Bayesian objective enables efficient and effective training without surrogate losses.
• The approach provides strong generalization performance and simplifies training schemes for specific network types.